Category Archives: Sisense
Data Visualization and Visual Analytics: Seeing the World of Data
Everyone wants to get more out of their data, but how exactly to do that can leave you scratching your head. Our BI Best Practices demystify the analytics world and empower you with actionable how-to guidance.
In a world increasingly dominated by data, users of all kinds are gathering, managing, visualizing, and analyzing data in a wide variety of ways. One of the downsides of the role that data now plays in the modern business world is that users can be overloaded with jargon and tech-speak, which can be overwhelming.
Data visualization and visual analytics are two terms that come up a lot when new and experienced analytics users alike delve into the world of data in their quest to make smarter decisions. In this post, we’ll dig into what these concepts are, their strengths, and how they work together.
Data visualization: painting a picture of your data
Simply put, data visualization means showing data in a visual format that makes insights easier to understand for human users. Data is usually visualized in a pictorial or graphical form such as charts, graphs, lists, maps, and comprehensive dashboards that combine these multiple formats.
The primary objective of data visualization is to clearly communicate what the data says, help explain trends and statistics, and show patterns that would otherwise be impossible to see. Data visualization is used to make the consuming, interpreting, and understanding data as simple as possible, and to make it easier to derive insights from data. When BI and analytics users want to see analytics results, and learn from them quickly, they rely on data visualizations.
Visual analytics does the “heavy lifting” with data, by using a variety of processes — mechanical, algorithms, machine learning, natural language processing, etc — to identify and reveal patterns and trends. It prepares the data for the process of data visualization, thereby enabling users to examine data, understand what it means, interpret the patterns it highlights, and help them find meaning and gain useful insights from complex data sets.
Analytics and visualizations, the key to data-driven organizations
The relationship between data visualization and visual analytics is symbiotic. In a study for the US Department of Homeland Security, James J. Thomas, and Kristin A. Cook illustrate this relationship in what they describe as “The Sense-Making Loop” of the analytical reasoning process:
Good data visualization enables visual analytics to be more effective and to show users better insights and better insights make for more compelling visualizations. Combining the two into visual data analysis makes it easier for users to better understand their data. Together, they help organizations and individuals identify how they can be more efficient, drive revenue, and gain a competitive advantage over their competitors.
The role of visualizations in analytics
Data visualization can either be static or interactive. Static visualizations provide users with a single view of what’s in front of them. Interactive visualizations enable users to drill down into data and extract and examine various views of the same dataset, selecting specific data points that they want to see in a visualized format.
Data visualization is what provides clarity to data-driven insights and it’s what enhances understanding throughout an organization.
In this diagram, visual analytics is shown to be the foundation for interactive data, thereby demonstrating how the two are connected. Analytics acts as the source for data visualization and contributes to the health of any organization by identifying underlying models and patterns and predicting needs.
Visualizations: past, present, and future
Broadly, there are three types of analytics: descriptive, prescriptive, and predictive. The simplest type, descriptive analytics, describes something that has already happened and suggests its root causes.
Prescriptive analytics takes things a stage further: In addition to helping organizations understand causes, it helps them learn from what’s happened and shape tactics and strategies that can improve their current performance and their profitability. A simple example would be the analysis of marketing campaigns.
Predictive analytics is the most beneficial, but arguably the most complex type. It helps users to identify patterns that suggest future situations and behaviors. Using predictive analytics, organizations can plan for forthcoming scenarios, anticipate new trends, and prepare for them most efficiently and cost-effectively. Predicting forthcoming trends sets the stage for optimizing the benefits your organization takes from them.
Using visualizations to make smarter decisions
The data drawn from power visualizations comes from a variety of sources: Structured data, in the form of relational databases such as Excel, or unstructured data, deriving from text, video, audio, photos, the internet and smart devices. This data is gathered into either on-premises servers or increasingly into cloud data warehouses and data lakes. They are transformed into data visualizations and shared via dashboards and analytic apps so that users can make smarter, data-driven decisions.
Data teams and business and analytics teams are tasked with choosing and developing the best way to visualize data and to build well-organized dashboards in order to help end-users make smarter decisions. Dashboards need to be clear, quick to interpret, and easy to drill into to find the deeper insights when needed.
To achieve this successfully, you need a data and analytics platform that offers a powerful combination of visual analytics and data visualizations; with the capacity to handle huge volumes of data either stored on-premises, in the Cloud, or both; with the flexibility to integrate data from any source; and with the scalability for future growth.
To achieve this successfully, you need a data and analytics platform that offers a powerful combination of visual analytics and data visualization; with the capacity to handle huge volumes of data either stored on-premises, in the Cloud, or both; with the flexibility to integrate data from any source; and with the scalability for future growth.
Visual analytics and data visualizations in action
Orion is a premier portfolio accounting SaaS provider for financial advisors. Previously, they had been using a static reporting solution that served up stale news with a time lag of weeks, while their customers wanted pre-configured, dynamic dashboards. By switching to Sisense, they were able to offer all of this, doubling sales revenue at the same time.
And in the fight against COVID-19, Resconsortium has used Sisense technology to map the spread of the virus across the UK on a dashboard. It provides local and regional National Health Service capacity planners with the real-time information they need to target resources to areas where the outbreak becomes more severe or where there is a higher density of at-risk patients. And the data is as granular as the patient lists at individual family doctors’ surgeries.
To get the best insights from your data, and to optimize the benefits from your BI and analytics, you need a seamless combination of visual analytics and data visualization. Both are important, but each can’t be as effective without the other. Together, they play a vital role in the analysis and understanding of your data, and your ability to shape a successful future strategy for your organization using the insights that they reveal.
Julie Zuckerman is a senior product marketing director at Sisense, bringing over two decades of experience in marketing and product marketing at tech companies. Her debut novel, The Book of Jeremiah, was published in 2019.
Five Design Tips to Deliver a Seamless Embedded Analytics Experience
In Managing Up, we give product managers and their teams actionable guidance on how to build, test, and release programs that will delight users and stand the test of time. From development to execution, our industry experts and seasoned pros can help you work smarter, not harder.
Delivering additional value to your customers by providing data-driven insights is a sure-shot way to improve your competitive advantage and monetize your data. It’s not a question of if but when this functionality becomes basic table stakes.
Several steps are required to turn data into revenue and go even further — including creating the right data strategy and identifying data opportunities, building your profit and loss statement and use cases, getting buy-in, finding a partner, implementing and launching, and finally growing post-launch through an iterative innovation process.
Zooming in on design and implementation: Delivering value means combining the right features, KPIs that drive action, and visualizations that tell a compelling story.
In addition, there is an important but oft-overlooked aspect to successful data monetization: user interface (UI) design (or visual design). While UI design is a vast topic, a common challenge when delivering analytics to customers is to create a consistent and seamless experience, since you are embedding a third-party application in your product.
Simple UI is good UI
Consistent design is intuitive design. It should be immediately obvious to the user how to accomplish whatever they want to do. Expectations of users today are higher, so you also need a beautiful design that intuitively directs a user to take action.
From a branding standpoint, it should also always be clear to the user that the product is unified: Typography, logo, image styles, brand color schemes, etc., should be reflected across the application (in the core offering and the analytics), just like the rest of the brand’s properties.
When embedding analytics, you are bringing two different applications into one space, so it is critical that your end-user seamlessly experiences both the core product and analytics.
Let’s take a deeper look.
Tip 1: Maintain consistent branding elements
Your brand matters! Consistent visuals resonate more effectively with your audience.
This is no different when you are delivering analytics to your customer. Remove any trace of third-party applications by changing any external logo or text to align to your organization’s brand. Leveraging a drag-and-drop UI-based interface to change the settings makes this process faster and will allow you to get to market rapidly without worrying about custom development.
Tip 2: Consider visual hierarchy
Colors can convey powerful messages, so it’s crucial they be used mindfully and consistently. It can be unharmonious to have your brand colors be teal and black, for example, and part of your application or the analytics you deliver be yellow and white.
Leverage your corporate brand guidelines, if they exist, to customize the analytic application’s primary brand colors, primary and secondary text colors, etc.
If brand guidelines don’t exist, use easily available color palette selectors (there are tons of options to choose from) to select a color palette. Build your palette with a primary color in mind (usually your main brand color). Most products that are well designed use a visual hierarchy of fonts and color palettes to help ignite the experience.
In the palette, make sure to have a couple of options:
- Primary brand color
- Highlight or accent color for banners or toolbars
- At least two light and two dark colors for contrast between background and texts (dark background, light texts or vice versa)
Once you have the colors, you can add them to “favorites” and make it easier to pick and choose.
In addition to the application colors, also make sure to change the visualization color palette to match the application brand color palette. You can either use one of the several out-of-the-box color palette options or create a custom visualization palette. Again, you can customize your palette to match your brand colors.
Tip 3: Build branding with typography
Typography is a crucial element that uplifts a design and gives it a personality. Typography is also an important element that should be part of your brand. Typography conveys personality and grabs a user’s attention while establishing the tone of your brand. Certain fonts can also improve scannability, legibility, readability, and even navigation. It’s important that the font or fonts you use are consistent throughout your digital experiences.
Tip 4: Minimize the steps to insights
It’s a fundamental tenet of UI/UX design that the fewer clicks a given action takes, the better. When embedding analytics, be sure to keep actions like filtering, exporting, etc., in one place in the host application so users don’t have to repeat the actions twice.
Think like a user: Would you want to first filter on the host application and then repeat the action again in the embedded analytics? Of course not. A seamless user experience (UX) means that when you do something once, it affects the entire application without you having to repeat steps.
Explore how you can implement these experiences and get inspired in the Sisense Embedded Playground.
Tip 5: Focus on ease of implementation
While good UI design makes a difference, it’s also important to be mindful of the cost, effort, and complexities of implementing seamless and consistent experiences.
Customizations that require developer skills and coding take considerable effort and increase time to market. These are also difficult to maintain over time. For example, upgrades can become a hassle due to backward compatibility issues. Changes to underlying infrastructure can break the code, requiring complex updates.
To avoid such hassles, find an analytics solution that makes rebranding and customization quick and easy. Using a solution that allows you to implement your changes via drag-and-drop, out-of-the-box capabilities that are always up to date with newly released versions helps you avoid some of these larger issues down the line. It also allows you to focus your efforts on data insights and application functionality instead of worrying about code, maintenance, or changes that can break your code.
Delivering standalone analytics solutions
Whether you choose to deliver analytics as a standalone product or service, or you embed them into your application, your analytics product should be an extension of your brand and visual story, and all the tips above remain true.
By white-labeling and customizing the look and feel of your analytic application, you deliver a consistent and great user experience! Sisense is equipped to allow you to put analytics where you need them and make them look and work the way you want. Whether you are creating your own solution from scratch or using ours, keeping these design tenets in mind will help you build an awesome product.
Shruthi Panicker is a Sr. Technical Product Marketing Manager with Sisense. She focuses on how Sisense can be leveraged to build successful embedded analytics solutions covering Sisense’s embedding and customization capabilities, developer experience initiative and cloud-native architecture. She holds a BS in Computer Science as well as an MBA and has over a decade of experience in the technology world.
Improve Your Product Differentiation with Embedded Analytics
In Managing Up, we give product managers and their teams actionable guidance on how to build, test, and release programs that will delight users and stand the test of time. From development to execution, our industry experts and seasoned pros can help you work smarter not harder.
Marketers are constantly communicating to consumers the advantages of their brand’s products—whether faster, cleaner, stronger, or cheaper. Having your product or service stand out in the consumer’s mind is the goal of a product differentiation strategy. If your products are not interchangeable commodities, such as sugar or soybeans, emphasizing the unique qualities of what you’re selling should result in less competition and more “stickiness” with your customers. This is where product teams come in: creating value for customers with unique selling points (USPs) that aid in product differentiation.
Horizontal vs. vertical product differentiation
One of the earliest product differentiation definitions came from the American economist Edward Chamberlain, who noted in his 1933 work, The Theory of Monopolistic Competition, that certain product features are more important than others to some consumers. Later, author and academic Michael Porter stated in the July 1980 issue of the Financial Analysts Journal that when a product is not a commodity, product differentiation “creates layers of insulation against competitive warfare because buyers have preferences and loyalties to particular sellers.” In other words, having a differentiation strategy makes business sense.
There are two ways that you can differentiate your products: horizontally—i.e., within the product line—and vertically, against external competitors.
Horizontal product differentiation is when a product stands out based on subjective characteristics. For example, there is no implicit benefit to having a blue Ford Mustang rather than a red one. The choice is purely subjective and often contained within the brand’s product line.
But while the color of the vehicle is subject to consumer preference, there are objective differences that a car company can highlight to set their vehicles apart from the competition. This is known as vertical product differentiation.
In a 1986 article for the American Economic Review, economist John Sutton noted that, with respect to vertical product differentiation, “if two, distinct products are offered at the same price, then all consumers prefer the same one (the higher-quality product).” However, the reality is that improved quality often results in an increased cost to the consumer. And the willingness of a subset of customers to pay for an objective increase in quality is what drives a company to vertically differentiate their products. So for sports cars, vertical product differentiation examples would include:
- Increasing horsepower
- Decreasing time from 0 to 60 mph
- Reducing carbon dioxide emissions
- Implementing environmentally sound manufacturing practices
- Manufacturing the vehicle in a specific location (e.g. “Made in America”)
In nearly all sectors, different attributes can sway different subsets of consumers. For example, one consumer may put the greatest weight on the environmental footprint of their vehicle (both in terms of emissions and manufacturing processes), while another individual may just have a “need for speed.” The key for product managers and product marketers is to target the unique qualities that consumers are willing to pay a premium for and emphasize those differentiators in the marketplace. That’s where an embedded analytics solution can help.
Embedded analytics: Driving product differentiation
For software businesses, analytics is often the differentiator between your application and the competition’s. The software-as-a-service (SaaS) market has exploded with the advent of cloud computing and the market has become crowded. Having embedded analytics built into your platform is one way to stand out.
Case in point with global supply-chain giant CTSI-Global. Their small analytics team helps provide insights to their clients and empowers end-users to build customized dashboards. Certainly, CTSI-Global would subscribe to the belief that every company is becoming a data company, and Sisense has been key in that transformation.
“The almost boundless agility of Sisense, combined with an efficient process framework, let us deliver and organically scale a market-differentiating product,” said Todd Winton, Development Manager at CTSI-Global in a recent case study.
For another example, look at Erea Consulting, a company that has solved legacy inefficiencies in retail data sharing between suppliers and retailers by delivering real-time insights. The information has improved suppliers’ resource allocation and promotional efforts, helping their products to fly off the shelves. Retailers can sell back data insights to suppliers, and thanks to Erea’s implementation of Sisense, some retailers have quadrupled their revenue from data.
Learning and growing with embedded analytics
Analytics aren’t just a way for Product Teams to set their apps apart from the competition in a crowded market, they’re also a key way builders can learn about their target audiences and users.
When browsing a brand’s website or using an app, prospects and current users always leave a digital breadcrumb trail that can yield valuable insights. Historically, the data in that trail would have to be pulled from a database and recorded in a spreadsheet to derive actionable insights. Not only was this time consuming, it was also prone to error, often leaving product teams without a single source of truth. But a data and analytics platform, such as Sisense, allows product managers to make data-driven decisions without having to wrangle the data themselves because all the information is in a unified view.
Those product-level decisions can relate to anything from color to product bundles that encourage a shopper to make the purchase. BraunAbility, the market leader in wheelchair accessible vehicles and commercial lifts, is one company that has used embedded analytics to better understand customer preferences and to ultimately increase profit margins across the board. Reports that once took weeks for analysts to derive are now available on demand, helping the company with horizontal product differentiation that’s driving results.
Setting your product apart with embedded analytics, easily
Maybe you’re a technology company looking to stand out from the competition by delivering actionable insights from your customer’s data. Soon, product managers will be confronted by the age-old decision: build versus buy. But, the decision should be an easy one.
Attempting to develop a homegrown embedded analytics solution within your technology may not be your best move, since this means taking your development resources away from their core duties building your product. Instead of building your own embedded analytics, partnering with a provider like Sisense can help you save time and resources and see ROI faster. Then engineers and product leaders at Sisense live and breathe business intelligence and have developed a high-quality, embedded analytics solution that’s been included in the 2020 Gartner Magic Quadrant for analytics and business intelligence platforms.
Teaming up with a partner like Sisense makes adding embedded analytics to your product super-simple. The Sisense end-to-end BI platform empowers your builders — those with natural curiosity, a penchant for asking deep questions, and a passion for serving up deep insights to users — to create analytic apps that deliver highly interactive user experiences and powerful data. Watch the demo now and find out why Gartner has labeled Sisense as a visionary in its Magic Quadrant for analytics and business intelligence platforms.
Eitan Sofer is a seasoned Sisenser, having spent the last 13 years building and shaping our core analytics product, focusing on user experience and platform engineering. Today, he runs the Embedded Analytics product line which powers thousands of customers and businesses, making them insights-driven. Eitan is also an avid music fan and surfer.
The Need for Speed: Faster Data Access as Competitive Edge
The cloud isn’t the future; it’s right now. In the Clouds is where we explore the ways cloud-native architecture, cloud data storage, and cloud analytics are changing key industries and business practices, with anecdotes from experts, how-to’s, and more to help your company excel in the cloud era.
The world of data is constantly changing and speeding up every day. Companies are storing more types of data from applications as well as the Internet of Things. This data flows into cloud-native warehouses where data teams manipulate it, allowing analysts to derive vital insights from it, and product teams embed those insights into products. Data is the bedrock on which the future of business is being built.
As the data that these businesses need to thrive continues to grow and its pace of change accelerates, it’s never been more important for employees at all levels of an organization to have fast access to actionable data in order to make strategic, operational, and tactical decisions. It’s both basic table stakes for success in the “new normal,” as well as a defining edge that companies can use to stay ahead of the competition.
Saving lives in real-time
Easier access to fast-updating datasets isn’t just about making better decisions or powering the next killer app. It also can also save lives and change the way the world works.
“There are a wide range of scenarios where having super-fast access to real-time data can make a huge difference,” said Christelle Scharff, a professor and computer scientist based at Pace University in New York. “Fast access to data captured by video surveillance systems, for example, can improve security… It’s also the driving force behind autonomous cars. Our biggest industrial firms can use it for preventative maintenance — saving potentially millions of dollars. And almost all organizations can use it to avoid potential threats from security breaches and malware attacks.”
The success of COVID-tracing efforts will depend on fast access to multiple data sources.
George Thiruvathukal, professor of computer science at Loyola University
During our current pandemic, access to real-time data can also save lives. “Health officials are investigating how contact-tracing apps can help manage the ‘reopening’ after we begin to reopen the country after the COVID-19 lockdown,” said George Thiruvathukal, professor of computer science at Loyola University in Chicago. “The success of this will depend on fast access to multiple data sources.”
There’s an impact on customer expectations too. According to recent research from IDC, consumers are embracing personalized real-time engagements and resetting their expectations for data delivery. As their digital world overlaps with their physical realities, they expect to access products and services wherever they are, over whatever connection they have, and on any device. They want data in the moment, on the go, and personalized. As a result, IDC predicts that nearly 30% of the global datasphere will be real-time by 2025.
Pressure on infrastructure builds
As enterprises demand data infrastructures that can meet this growth in real-time data — and ultimately assist with their product differentiation strategy — the pressure put on product teams is huge.
Product teams are already having to manage the growing complexities that come with modern data environments.
Chandana Gopal, Business Analytics Research Director, IDC
“Product teams are already having to manage the growing complexities that come with modern data environments,” said Chandana Gopal, research director for business analytics at IDC. “Not only do they have to deal with data that is distributed across on-premises, hybrid, and multi-cloud environments, but they have to contend with structured, semi-structured, and unstructured data types. Multiple technologies to manage data at rest and in motion have compounded the challenge of managing data and making it accessible to decision-makers in the right time, in the right format, and in the right context.”
Managing cloud data is a key challenge for data and product teams who are tasked with connecting to a wide array of datasets stored in cloud-native warehouses and other locations. In a large-enough company, there can even be multiple clouds being operated by different divisions and teams. BI and analytics providers have had to design their platforms to serve up fast insights no matter where the data being analyzed resides, even partnering with third-party companies to make sure that their platforms can handle data from oft-used services like AWS, Google Cloud, and others.
Which makes sense, as customers searching for an analytics solution are also often grappling with their recently-purchased or possible cloud options:
“When customers come to us for their BI and analytics needs, in the same sentence they’re often telling us that they’re considering their cloud options,” said Erin Winkler-McCue, Lead for Strategic Partnerships & Special Projects at Sisense. “These conversations are no longer siloed. Customers want to know that our platform will work seamlessly with their chosen cloud vendor, even if that just means something as basic-sounding as making sure queries between the vendor and Sisense are optimized.”
The challenges these teams face become even more daunting when one looks towards the future, as new technologies like the internet of things, machine learning, 5G, and augmented reality will add a new level of demand. Forbes Insights data shows that in order to benefit from emerging technologies like these, 92% of CIOs and CTOs say their business will require faster download and response time in the near future. What’s concerning is that, despite recognizing this, just 1% of data center engineers believe their data centers are updated ahead of current needs.
Competing priorities within companies
All of this results in a lot of friction within data-driven organizations. “Multiple technologies are required for managing, integrating, and controlling the flow and consumption of data from the edge to the cloud and all points in between. That’s without mentioning outdated metadata—the data about data that provides data intelligence,” said Gopal.
An upcoming skills gap might compound the problem: According to the Forbes Insights research, 37% of engineers say they will likely retire in the next 10 years.
Adding to this hurdle is the fact that some firms are led by executives that don’t understand or champion the importance of having contextual and timely data embedded into applications. Recent research by Exasol has found that less than half of decision-makers believe that those working in senior management (40%) or mid-management roles (32%) are very effectively informed of their organization’s data strategy.
Creating a path to success
Gopal believes that future success requires that data teams take a structured approach focused on people, processes, and technology in order to make data available to all.
“Data teams should identify short- and long-term data and analytics use cases that will demonstrate business value with input from stakeholders at all levels—both business and IT,” she said. “They should also identify data-related assets that will be required for the project and be realistic about time constraints. They should then look to deliver measurable value with short term projects to build business cases for more expensive or longer projects.”
Teams should look to deliver measurable value with short term projects to build business cases for more expensive or longer projects.”
Chandana Gopal, Business Analytics Research Director, IDC
From a technology perspective, the introduction of new technologies, such as 5G-enabled edge computing, will have an impact on IT staffing. According to the Forbes Insight report, almost three-quarters (74%) of C-suite executives believe staffing will be reduced or handled by external cloud or edge service providers. The ability to implement new technologies like these in the data center will be a competitive differentiator, as will better security (according to 43% of respondents), and bandwidth (according to 27%).
Ultimately, it’s clear that organizations need to act quickly if they want to succeed. “Continuous efforts to update the data center will be integral to business success,” states the Forbes Insight report. “Partnering with external third parties is a central part of the data center journey in the age of hyper-connectivity.”
These collaborations are happening between companies and their cloud providers and between platforms like Sisense and companies like Amazon and Google:
“Partnerships with companies like AWS, Snowflake, Microsoft, and Google are only becoming more important as the modern data landscape evolves,” said Erin Winkler-McCue, Lead for Strategic Partnerships & Special Projects at Sisense. “We feel like every customer is either already in the cloud or it’s only a matter of time until they contemplate setting up a hybrid- or full-cloud model.”
Gopal agrees that adopting new technology through new partnerships is key: “A new class of intelligent data operations platforms are emerging that can reduce friction, improve efficiencies with automation, provide flexibility and openness with policy and metadata-driven processes that can accommodate the diversity and distribution of data in modern environments,” she said. Equipped with these, product teams will be much better prepared for a new and exciting future.
Lindsay James is a journalist and copywriter with over 20 years’ experience writing for enterprise business audiences. She has had the privilege of creating all sorts of copy for some of the world’s biggest companies and is a regular contributor to The Record, Compass, and IT Pro.
A Guide to the Best Javascript Visualization Libraries
Developing analytic apps is a bold new direction for product teams. The Toolbox is where we talk development best practices, tips, tricks, and success stories to help you build the future of analytics and empower your users with the insights and actions they need.
JavaScript data visualization tools are in greater demand now than ever before because of the enormous growth of data. Marketing, finance, and sales teams all rely on visualizations to help them understand their data. Javascript visualization libraries empower product teams and other builders to create custom visuals that support their partners and can even be used to create new revenue streams in the form of customer-facing analytic apps.
Whatever their specialty or goals, users want their data to be displayed visually so that they can see insights from their data easily and act on them quickly. Data visualizations of key performance indicators (KPI) can even be sent automatically to users as they change, allowing them to make faster, smarter decisions.
Understanding Javascript data visualization libraries
JavaScript data visualization libraries don’t just create beautiful visuals, they help make business intelligence (BI) more real to end-users. The developers building these visualizations must be guided by their colleagues who will actually be making use of them. They must also have access to flexible BI platforms (like Sisense) that can handle visualizations from a wide array of Javascript visualization libraries, which will allow them to input custom visuals that show users what they need to see in a way that makes sense to them. These visuals can include:
- Customizable charts — sunburst, Sankey, dependency, and others
- Flexible API with clear documentation and sample code
- Vibrant chart animation capabilities
- User interactivity with chart elements
- Compatible with React, Angular, Vue, and other libraries
- Chart/Data export to diverse formats: PDF, JPG, PNG
- Multiple data type compatibility: XML, JSON, CSV
Of the 50 or so competing JavaScript visualization libraries available, we’re going to explore 5 libraries that deliver a wide array of features to try to help you determine which one is the best for your application.
This decision may be dictated by a single factor such as compatibility with an existing Angular app or by an end-user performance requirement such as minimum chart rendering time. We’ll explore the intricacies of the different JavaScript libraries by reviewing example charts and code snippets.
Stunning chart diversity with D3.js
The D3.js data visualization library supports an impressive array of chart options. Combining user interactivity with chart animation puts this library in our top five. The best place to start comparing the capabilities of visualization libraries is to look at example charts and the code that generates their elements. So, let’s jump right into D3.js with a popular sample chart from this library’s GIT repo.
This example chart interactively reveals the hierarchy of relationships among the classes of an application. In this screenshot of the interactive animation, hovering the mouse over “Data” lights up the functions shared among the classes. In this scenario, the class functions which it imports highlight as red curves, while classes which import its functions are shown as blue curves:
When reviewing the best visualization libraries, it is important to be aware that many other libraries are actually built with D3.js. For example, Nivo, Plotly.js, and Recharts are all built from the open-source D3.js library.
The D3.js API is flexible and features the popular d3.js dashboard which is also widely implemented in the best analytics applications — apps which demand the best quality chart performance. This code snippet contains the function which maps the hierarchy in the above chart (source):
function hierarchy(data, delimiter = ".") {
let root;
const map = new Map;
data.forEach(function find(data) {
const {name} = data;
if (map.has(name)) return map.get(name);
const i = name.lastIndexOf(delimiter);
map.set(name, data);
if (i >= 0) {
find({name: name.substring(0, i), children: []}).children.push(data);
data.name = name.substring(i + 1);
} else {
root = data;
}
return data;
});
return root;
}An impressive selection of D3.js chart types, along with interactive examples and code snippets, is available at the D3.js GIT repo.
Google Charts — all the basics
Featuring a comprehensive chart gallery and extensive documentation of the API, Google Charts fulfills the expectations one might have when “standing on the shoulders of a giant.” The Chart Gallery contains examples of 29 chart types with example code for developers who are getting started charting with JavaScript. Since we began on page one with a D3.js Sankey diagram, let’s compare that with Google’s Sankey:
Google Charts’ version has glowing color mouseover interactivity with popup comments and a rich set of features to bring the chart to life and keep the user focused on critical data. Here is the sample code to define the colors in the chart (source):
var colors = ['#a6cee3', '#b2df8a', '#fb9a99', '#fdbf6f',
'#cab2d6', '#ffff99', '#1f78b4', '#33a02c'];
var options = {
height: 400,
sankey: {
node: {
colors: colors
},
link: {
colorMode: 'gradient',
colors: colors
}
}
};Google Charts documentation is concise and clearly organized. It’s also is free to use, but not open source. Data must be piped to the Google Charts API for plotting. The implication here is that users may not be comfortable with the liability of shipping confidential client data out of the environment when the Google source can’t be hosted in-house.
Highcharts: Strong community and API reference
A robust charting library, Highcharts has both free and paid versions. The source code can be installed in-house or in a Cloud PaaS, which is advantageous for analytic apps with strict data security constraints. A significant set of large enterprises use Highcharts, and so the community of contributing users is large. Beyond the opening list of standard features, Highcharts’ high points include:
- Excellent API reference, and community showcase.
- Compatible with Angular, React, and .NET plus more.
- Charts export to all web image formats.
Let’s look at an interactive Highcharts example. A variation on the Sankey chart concept, the dependency wheel is also a flow diagram. However, the nodes are mapped to the edge of a circle with the links connecting the nodes. As with the Sankey, the width of the links and size of the nodes are proportional to the quantity of flow. Unlike the Sankey, though, the dependency wheel flow is multidirectional and single-layered. In the following example, a country’s export dependency on other countries is highlighted on mouseover event:
amCharts: Great for touchscreens
A distinguishing feature of the amCharts JS visualization library is specialized touch screen interactivity for charts displayed on mobile devices. This will appeal to endpoint users such as sales engineers who want to monitor product performance quickly while in the field. Zoom charts which illustrate more data at deeper levels can now be explored readily on tablets and phones.
amCharts is compatible with all the major frontend development frameworks like Angular and React. The amCharts Sankey diagram is configurable for animation. Here is an example with a distinctive color scheme which shows how the relative size of nodes and links illustrates quantitative proportionality in this chart of tech revenue distribution.
Although amCharts’ primary product is a paid version, it is possible to use the library for free on the condition that all generated charts contain the amCharts logo. So, this will not work for companies offering a whitelabel analytics app. There is a rich set of demos of all supported chart types with complete documentation and code samples, all of which lends well to the customization of charts.
Recharts: built for React
Although Recharts is actually built from D3.js components, it is important to our discussion for two reasons. First, Recharts is specifically intended for use with React, as essentially all of the D3.js library is wrapped in React classes. Secondly, Recharts is very popular, boasting a number of downloads not far behind Highcharts. Now that we are operating in the age of ubiquitous white-labled web apps, many React developers will find themselves using Recharts without knowing that they’re actually using the D3.js core.
Another thing to note about the Recharts is that there are two significant disadvantages to using it. For one, there are substantial open issues on the GIT repo, and bugs seem to appear faster than they are resolved. Additionally, Recharts can only be used in React projects. The large number of downloads also speaks indirectly about the wildly popular React JavaScript framework. This paradoxical limitation to React projects likewise refers back to our initial point that the choice of visualization library may be dictated solely by the application’s pre-existing development framework.
Choosing the right JavaScript visualization library for you
Having surveyed five of the most popular JavaScript data visualization libraries, you can see that there are substantial differences. Can your infrastructure host the library locally? Is the library license paid or free and does that depend on the intended use? Here is a list of basic criteria to consider when making the decision:
- Type of charts required: Gantt, Sankey, Dependency wheel.
- Level of support for library customization.
- Size and characteristics of dataset
- Browser support for library of choice: Chrome, Firefox.
- The JavaScript framework used for the app: Angular, React, etc.
- Intended device: desktop, mobile, or both.
And yet there is still one more very important criterion for choosing the right visualization app: are you adapting the library for use with an analytic app? For example, Sisense Analytics can be extended to integrate with any javascript visualization library using our JavaScript Add-On Framework.
Detailed documentation examples and code snippets are crucial for the quick and easy library usage. You can find both in the official Sisense marketplace and Community-Led Add-Ons. Sisense implements a wealth of expert domain knowledge in the field of data visualization to generate a ton of actionable insights on live streaming data.
Eitan Sofer is a seasoned Sisenser, having spent the last 13 years building and shaping our core analytics product, focusing on user experience and platform engineering. Today, he runs the Embedded Analytics product line which powers thousands of customers and businesses, making them insights-driven. Eitan is also an avid music fan and surfer.
How to Build an Outcomes-Focused Data Strategy
Money never sleeps and neither does your data. In Monetizing Your Data, we look at digital transformation: the ways of turning data into new revenue streams and apps that boost income, increase stickiness, and help your company thrive in the world of Big Data.
The first waves of digital transformation started decades ago and the ripples of this trend continue to be felt to this day. However, what exactly a digital transformation looks like varies widely from company to company. One common theme among many transformations, however, is trying to make better use of data, whether to build analytic apps to unlock new revenue streams or to make smarter decisions internally (or both).
While these are worthwhile applications, one blind spot that many teams charged with these projects share is that they look at the data they have on-hand before figuring out what kind of problems they wish to solve with it.
“I recommend starting your data strategy with a right-to-left approach, focusing on the desired business outcomes first, instead of the data, to support those outcomes,” says Charles Holive, Sisense Managing Director of Data Monetization and Strategy Consulting. “And there are primarily three areas that industries across the world look to improve: the top line, the bottom line, and customer satisfaction.”
Define your desired outcome — before you start building
Every company knows they need to digitally transform in order to survive and excel in the modern era. However, many organizations fail to define their goals for this process before they start, and predictably encounter obstacles or outright failures instead of paving a path for future success.
Business goals should be defined at the very beginning of the digital transformation in the “right-to-left strategy” that starts by answering this question: What is the organization specifically looking to solve or improve? Understanding the details is key, otherwise “digital transformation” will be merely a corporate buzzword that causes headaches, heartbreaks, and lost money instead of producing measurable improvements.
From there, rather than trying to accumulate and house the company’s entire dataset, the digital transformation team should identify the specific actionable insights and associated data needed to solve for (and measure) agreed-upon outcomes.
“Not every dataset is made equal; some are more valuable than the others. So being outcome-focused is a way that can you stack-rank the data sets that are most important. Your team can then begin moving that most-important data into your warehouse.”
Experiment to guide a winning data strategy
Just as the waterfall method of software development—the strategy of gathering all the requirements upfront and then building and releasing a complete application—has fallen out of favor for agile methods, the same thing should happen when creating an outcome-first data strategy: Rather than trying to build a complete data warehouse right from the outset, approach data strategy as an “innovation factory.”
“Identifying the exact data you need to solve a singular problem results in a perfect candidate to go into your warehouse on the first cycle. This is because you know exactly what the business is going to do with that data set,” Charles explains. “It’s powerful because it’s already informing or measuring a specific business outcome.”
And when this data is warehoused and accessible to business partners to make key decisions, you already have a chance to quickly prove this outcome-first data strategy. You’ve immediately created an experiment to win.
Another piece of advice that Charles talks about in his “Hacking the Analytic Apps Economy” video series is where the innovation factory should live. Namely, not in a mature business unit, but in an agile, fast-reacting department that reports to a Chief Innovation Officer or similar highly-placed exec. This team can deliver on new ideas quickly and won’t get bogged down in pre-existing frameworks or goals that don’t work for what the new data experiments are trying to achieve.
Create an innovation factory at your company
“Creating an innovation factory for your company results in faster innovation. You can do these smaller experiments more cost-efficiently, saving money over the traditional data strategy. This also should help your team prioritize projects for the data warehouse that deliver the greatest value, as opposed to the department that screams the loudest.”
And while any experiment can fail, but here are some solid tips to help improve your likelihood of success and to maximize the impact of triumphant experiments:
- Start by listening to the frontline employees who use the data to make decisions—this will improve the odds of success for your experiment out of the gate.
- If your experiment works, find other departments that can benefit from that same data—this is where it is key to have a good semantic layer on top of your data warehouse (courtesy of your data analytics software) so you can repurpose the same dataset for different ends.
- If your experiment fails, see if you can tweak the dataset or use case to apply elsewhere in the company.
Regardless, approaching data strategy with a focus on business outcomes will put you on the right course.
“Everything else in the company is business-centered. It just seems counterintuitive not to approach data strategy in the same way.”
Jack Cieslak is a 10-year veteran of the tech world. He’s written for Amazon, CB Insights, and others, on topics ranging from ecommerce and VC investments to crazy product launches and top-secret startup projects.
Living Post Uncertainty: How Changes Today Will Propel Future Generations
COVID-19 is the biggest human challenge we’ve encountered in decades — and despite the focus on epidemiology it actually centers on data. In Data Surrounds Us, we shift our focus from what happened to what we can do better. Through understanding past crises and recent data in the world around us, we want to help everyday people understand how their choices will lead to a better outcome, dispel uncertainty and use data to chart, draw, and paint a better future.
COVID-19 happened so quickly — and by now it is clear to us all that it will reshape our society in so many lasting ways. It shocked, awed, and changed us in just a few short weeks. While almost everyone was confined to their homes by government-mandated shutdowns, isolation led to deep questions about the world around us: How we’re governed, what matters to us, and with whom we want to spend our precious time. In short, this large, impersonal single-cell driven crisis feels oh-so personal to us all, and is creating in many of us a sense of hopelessness and helplessness.
This isn’t the first time that society collectively experienced a traumatic shock. Both the 9/11 terror attacks and the 2008 economic collapse are still fresh in our recent living memory, but the real comparable events occurred in 1918, 1957, and 1968. This was especially poignant when death totals from the 1918 Spanish Flu exceeded tens of millions of people. Only a handful of survivors of the 1918 pandemic remain, and thus our current crisis is new to nearly all of us.
Even taking into account the two other pandemics of the 20th century, very few people experienced similar levels of fear, uncertainty, and doubt. Past pandemics affected the world in significant ways — politics, lifestyle, families, and of course the economy. There are lessons from those prior events: During the actual lockdowns, but also in the years that followed and the people who endured those tragedies. How we respond will certainly color how we look back on this crisis.
Living in a world in flux
As we entered 2020, things were already changing — political upheaval, globalization breakdown, automation job replacement, innovation moving faster than human comprehension — the world was already in transition. This pandemic hit hard and fast, but now that the shock is subsiding, all nations must begin to chart towards a sense of semblance of normalcy. Many questions still remain unanswered so we should, therefore, note a key learning moment: This crisis provides us all with a chance to pause and reevaluate.
We have an opportunity to look at the data and to decide the world we want to live and work in, and rebuild it together for the better. In my day job, I serve as the Chief Strategy Officer of Sisense, a company that helps deliver data and analytics via an advanced technology platform. More importantly, our mission is to deliver insights to the builders of new constructs, be that data structures, analytics dashboards, or new businesses. Now more than ever, data will be a barometer of truth.
The last few weeks were an adjustment period, where people watched and debated data more than ever before. Hundreds of dashboards count and filter a local view of infections, deaths, and other key metrics around the coronavirus pandemic, and it impacts our collective psyche. This data is disorienting, morbid, and overwhelming but we are all figuring out how to survive this, together. But despite all of the negative signals, I believe, there is also hope.
Data will guide us towards a new normal
Data can contribute to the healing process, but also shines a bright light on new and exciting changes in our society. Data cannot replace our values, nor should we simply consign our societal decision making processes to a set of smart algorithms. However, we should use the cast light of data-driven knowledge to gain insights and outline avenues into the future we should strive to shape.
While it’s a jarring shift, many of the things we are doing today will inform a tomorrow that shouldn’t feel distant. These things include our increased focus on the environment, the changing way we are using technology, celebrating front-line workers, how we shop and purchase items, and countless other factors that impact our personal and professional lives.
We are capable of doing more for ourselves than we ever thought. We are resourceful and many people are returning to a simpler way of doing things — whether that’s staying at home with family or getting outdoors (safely) looking for healthier lifestyles. It changed the way we perceive how we could work, the way we may live, what and how much we should give, how our kids would learn, and even how we consume and process information.
In short, it impacts what we prioritize, how we create a sense of purpose in our lives, and where we invest our time. Reality has shifted around us and as we find a new stable perch for our minds in the new normal we can choose to direct our gaze, goals, and values to a new horizon, otherwise blocked by the pre-pandemic grueling daily momentum.
Introducing our COVID-19 Hub
Today, we’re launching a new initiative that explores the impact of COVID-19 on the world, showcasing data that predict a new, better world. We will curate and share trends, surveys, and insightful data. This “global timeout” allows us to realize the many things we can do without and the really important things we truly want back. Family quality time, self-development pursuits, reconnecting with friends, nurturing local initiatives, nature sorely missed, and many more. These are only a few of the newly rediscovered human life truisms which emerged during these days of forced seclusion. No matter how much the past might seem appealing, the world of 2019 is gone, and the future is now, and in our hands for shaping.
There are many questions to be targeted by human endeavor, but our world is embracing change more quickly than ever. At Sisense we want to help you curate and experience those changes through data tips and stories. As the world re-emerges, tough decisions will need to be made around the timing and structure of our transition back to normalcy. Some of these decisions will be generational (like the timing of gen-z vs. boomers returning to work) while others will impact everyone (like the timing of a vaccine rollout).
Data-driven decision-making is more important than ever
The decisions we make today will profoundly impact tomorrow for us, our children, and grandchildren. So, in the coming weeks and months, we’ll regularly share insights as a message of hope and with the goal of providing ideas around what a better future could look like. Data will be at the core of each of these stories because that is what is needed, and yes, also because that is what we do best. And the right data is free from political bias or gut feel; it’s something that is indisputable.
We hope you lean into the future with us, because we plan to push the limits around not only the ebbs and flows of the COVID-19 virus, but paint a picture around a pathway to what’s next. The future will be about a new purpose and a complete pivot that will be driven by analytics that empower the builders of tomorrow. We will each get to define our future.
Guy Levy-Yurista, PhD is the Chief Strategy Officer at Sisense, and an executive leader and entrepreneur with over 25 years of experience in Fortune 500, startup, and venture capital environments. Guy specializes in commercializing advanced technologies in dynamic market environments. He holds in-depth knowledge of analytics, social media, mobile, telecommunications, cyber-security, and optical technologies. As a thought leader and sought after speaker he is a contributing author to multiple media outlets and keynote speaker at a variety of industry and investor conferences.
How to Aggregate Global Data from the Coronavirus Outbreak
As COVID-19 continues to spread, healthcare groups and companies of all kinds are under pressure to provide care in the face of increasing demand. Healthy Data is your window into how data is helping these organizations address this crisis.
As the rapid spread of COVID-19 continues, data managers around the world are pulling together a wide variety of global data sources to inform governments, the private sector, and the public with the latest on the spread of this disease.
In this article, we discuss how this data is accessed, an example environment and set-up to be used for data processing, sample lines of Python code to show the simplicity of data transformations using Pandas and how this simple architecture can enable you to unlock new insights from this data yourself. Let’s get started.
The importance and impact of reliable data
Reliable data of this scale has helped to promote responsible decision-making in workplaces and communities around the world, including legislative action around international travel, the provision of emergency medical resources, the bolstering of financial markets, support for small business owners and proprietors, and medicine and treatment to those who are infected.
Households and families are also using this data to prepare for the nuances of everyday life that will change as a result of this virus, whether it be securing a paycheck, buying groceries and essentials, playing with their kids, inviting the neighbors over for a barbecue or walking the dog.
The importance of having properly-prepared data is paramount to the success of our attempts to mitigate and contain the spread of the virus, and the impact of information sharing at this level is truly transformative. But this type of globally aggregated data doesn’t just appear all on its own. Reliable sources need to first be discovered, retrieved, parsed, and aggregated together before they can be distributed around the world. This is not something we’ve been ready to prepare in such a short period of time until now.
Specific challenges involved with data related to the Coronavirus
Access to a reliable source of contagion data for a global event happening in real-time is not easy to discover. The major repositories that include data on public health issues or disease outbreaks can be accessed through an API.
Often the data is needed to be communicated to the public so quickly after it’s generated that it’s provided via more accessible methods first, leaving the programmatic access via the API for future historical analysis. These methods (PDFs or HTML tables) can be time-consuming to parse and scrape correctly and consistently across a bulk set of examples, yet that wasn’t enough to stop the developer community from getting started.
Since then, a team of researchers affiliated with Johns Hopkins University has been retrieving PDF files posted on the websites of these organizations via programmatic methods and parsing their contents into CSV files stored on a public repository on GitHub. This repository has been starred over 15k times and forked over 7k times. It is being used as source data for engineers around the world to ingest into their data pipelines.
In providing a globally useful dataset they have connected with over 14 data sources from around the world and aggregated them into their data model. Not all these data sources should be treated the same way, they each have specific needs. Some data sources include confirmed cases while others included presumptive cases. Some data sources are Excel spreadsheets or HTML tables while others are location pins on Google Maps. And some are in completely different languages than English.
The team at JHU has done a fantastic job in performing the bulk of the tricky normalization for us, yet they continue to face challenges every day in conforming to the new shapes, sizes, and formats of data that seem to be coming online exponentially as this virus continues to spread across our people. In the next section we’ll provide the link to access this data and perform some data cleaning and normalization operations using Python and Pandas!
How to set up your own data environment for analyzing COVID-19 data
Data Access
Data access is provided by Johns Hopkins University. They have built a pipeline that ingests global situation reports from the World Health Organization, translates data from South Korea, China, and Taiwan into English and has access to 10+ other global sources that appear to be quite labor-intensive to retrieve (a lot of web-scraping).
While some cleaning and normalization has already gone into this dataset, such as converting all timestamps to a UTC time zone and addressing some inconsistencies related to update frequency, there are still plenty of opportunities for us to dive in and focus on cleaning and normalization activities that will unlock real insights.
| # | Data Publisher | Scope | Source | Format | Update Cycle | Location |
| Johns Hopkins University* | USA | GitHub | JSON | Daily | https://github.com/CSSEGISandData/COVID-19 |
* Johns Hopkins University is actively parsing daily situation reports from WHO and integrating into their data model for the open-source community to access
Data infrastructure
Here is an example of a simple, cloud-based architecture that is suitable for rapid deployment of a data pipeline. In the environment created for this article, a Virtual Private Cloud (VPC) containing a Linux EC2 instance, a PostgreSQL database, and an internet gateway was spun up on Amazon Web Services and which was then connected to an external BI dashboarding tool.
The pipeline manager hosts a Python installation and Apache Airflow task scheduler (developed by Airbnb) which operates the data pipeline. After simple configuration Airflow is up and running and is executing Python scripts itself, writing data to the database every time the source data is updated.
Meanwhile, the internet gateway allows for external BI tools to connect to the data using a trusted connection so the data can be explored visually, and reports generated for communication of information to others. This is done without needing to download a local copy of the data. This architecture allows for data in the database and downstream reports to stay up to date automatically.
Processing and cleaning the data using Python and Pandas, writing to SQL database
And finally, we will go through a few simple examples of data cleaning and normalization performed in Python that can be used on this dataset in order to insert into a SQL table and query for valuable insights. The goal here is to format data sources into a common structure for effective bulk processing.
1. Combine data from multiple .csv files and dropping duplicates that may exist.
import pandas as pd
import os
df = pd.DataFrame()
for filename in files:
filepath = f'csse_covid_19_daily_reports/{filename}'
with open(filepath) as f:
da = pd.read_csv(f)
df = df.append(da, ignore_index=True, sort=False)df.drop_duplicates(inplace=True)
2. Fill null values with empty strings to prevent rows from being removed from subsequent table transformations.
df['Province/State'].fillna('', inplace=True)df['Country/Region'].fillna('', inplace=True)
3. Sort dates in descending order with most recent at the top.
df.sort_values(['Last Update'], ascending=False, inplace=True)
4. Convert datetimes to date and prepare to group and retrieve the most recent record from each date.
df['Last Update'].apply(lambda x: pd.to_datetime(x).strftime('%Y-%m-%d'))
5. Grouping and aggregating to retrieve the latest report from each day.
df.groupby(['Country/Region', 'Province/State', 'Last Update']) \
.agg({
'Province/State': 'first',
'Country/Region': 'first',
'Confirmed': 'first',
'Deaths': 'first',
'Recovered': 'first',
'Latitude': 'first',
'Longitude': 'first'})
6. Convert floating-point decimal fields to type “int.”
for col in ['Confirmed', 'Deaths', 'Recovered']:
df[col] = df[col].astype(int)
7. Write the data to a SQL database.
df.reset_index(inplace=True)
df.to_sql('cssc_daily_reports', con=connection, index=False, if_exists='replace')8. Connect to your SQL database and visualize your newly written data.
In this graphic we see that China’s cases mostly leveled off after 3-4 weeks, while growth in other countries is still ongoing.
In this graphic we see the differences in new cases each day between the United States, Germany, Italy, and Spain.
Fighting fire with data
As we continue to encounter global challenges at this scale, whether it be the coronavirus or another civilization-halting crisis, the importance of data collaboration across countries and state lines should not be underestimated. Not only because data wins debates, but because good data converts talk into consensus and action.
Govind Rajagopalan is a Senior Engineering Manager at Sisense. He has over 15 years of experience across both engineering and management roles with diverse companies, domains, teams, and technologies. He is excited to teach, help his teammates thrive, and have fun improving his craft.
Winning with Data: A COVID-19 Relief Package for Customers
Together, we face a truly unprecedented situation. The global Coronavirus (COVID-19) crisis affects all of our families, our businesses, our communities, and our very way of life. In times of great disruption, our collective ability to continue to march forward and identify solutions will ensure we persevere.
As a data and analytics company, we are doubling down on doing everything possible to help enable both those on the front lines who fight this crisis directly and those impacted by it across so many important industries, from small businesses to Fortune 100 companies. During this time, Sisense is doing everything in our power to support customers and the broader industries and markets that we serve.
Tackling new challenges in the age of COVID-19
Personally, I’ve spent most of the last couple of weeks talking (over video, of course) to our customers, as well as sourcing ideas from our employees about how we can help them. Dozens of customers shared their stories about new challenges that their businesses are facing. Our reality has changed — from WFH and social isolation to tele-everything and healthcare capacity challenges, to mass reductions of flights and ever-shifting supply chains. We live in a new normal that is unlike anything we’ve experienced.
I’ve spoken to a credit union using analytics to identify new unemployed members to provide direct support. I’ve spoken to a railroad that needs to redirect all of their routes to meet the new demand curve. I’ve met a healthcare company that is learning to optimize capacity and advance resource planning. I’ve met an eCommerce company using data to understand and adjust for a sudden burst of activity.
We’ve worked with a municipality whose elected leaders are using dashboards to communicate critical information for decision making, as well as a state that strives to balance the challenges of the crisis between ensuring both public health and economy, a balancing act that reviews different scenarios to make critical decisions. We’ve helped a company tracking the impact of the virus in the UK to guide citizens accordingly, and a NY entity that joined the NY COVID-19 task force.
Everyone is impacted. Data will be central to how many organizations rethink how to manage inventory, reallocate staff, optimize supply chains, or reassess marketing activities. Our customers need to respond quickly using data and analytics, as well as AI and machine learning to make better, smarter decisions. They cannot afford to wait months building new data warehouses or IT projects. They need nimble decision-making tools and empowered data teams.
Launching a COVID-19 Relief Package to Our Customers
We are incredibly grateful to the 2,000+ customers around the world who entrust us with their analytics needs. Sisense is a global company that is now fully remote and distributed across three continents. In addition to the many individual discussions and help we regularly provide, we’re formally announcing a comprehensive relief package for all customers to ensure their success and effective reaction to this crisis:
- Free unlimited analytics: Each and every one of our customers will now receive free licenses for uncapped, unlimited use of our analytics for 90 days to expand the use of Sisense to others within their organizations to provide insights that improve decision making.
- Free help/professional services: Data is at the heart of many of the most important business decisions, so we’re offering a few hours of help and technical consulting at no cost to address near-term business problems or conduct strategic planning projects. This can help take advantage of all the free licenses or be used for other initiatives.
- Strategic consulting with our leaders: Our experienced executive leadership team has built a library of best practices and tools they use to run their departments during this special time and shared it with all our customers. They are also available for business counsel, to guide our customers where analytics will provide the greatest impact.
Acting today for a stronger tomorrow— helping to empower those directly fighting COVID-19
Analytics plays an essential purpose on the world stage and provide great value during this crisis. Darwin once said, “It is not the strongest that survive, but the most responsive to change.” The world’s response to this crisis is extraordinary, as much as for what it has required from us as people to what it has revealed around how we work, and the impact we can each make.
We also want to contribute on a global scale. To those organizations who are on the front lines (committing their own resources, time, and energy to battle this crisis), we’re offering our software and professional experts at no cost to any organization engaged directly in the fight against COVID-19, to rid the world of this very real pandemic. We will make it free for the next 90 days as a start, and more as needed.
Our talented team at Sisense, as well as so many other good and resilient people, will rise to meet — and defeat — the challenges posed by this virus. If there are any critical issues or if you are aware of any opportunities for us to help others with our technology or analytics expertise, please let us know at [email protected].
I will continue to communicate with you around what else we are doing and know that we always value your questions, ideas, and feedback. We will overcome this challenge together.
Amir Orad is the CEO of Sisense, a successful entrepreneur and a Big Data, cybersecurity, and financial technology thought leader, with proven success in leading and scaling businesses by orders of magnitude.
Building Better: Ethics and Product Development
As software’s role in business proliferates, the ethical implications of decisions made by developers only become more profound. In Big Questions, we outline the biggest concerns facing development teams and look at how companies are grappling with them.
In our increasingly digital world, software has a fundamental role to play in almost everything we do. It’s the backbone of how our global workforce collaborates, it underpins much of our industrial operations, and even ensures the smooth-running of our cities.
A failure of these critical software systems, whether intentional or accidental, can result in damaging, even fatal, consequences. Unfortunately, these failures are all too common. It seems that every week we hear of another privacy infringement, security breach, or technology misuse in the news. With all this in mind, it’s never been more important for software developers to be aware of their ethical responsibilities.
Building a more secure future
The information means that a company’s data and infrastructure are its greatest assets, and weaknesses. According to Mandiant in their 2020 “M-Trends Report,” 22% of attacks on companies were for IP theft or corporate espionage, while a whopping 29% of attacks were for direct financial gain. Interestingly, only 4% of attacks covered by the report were to weaken the target’s systems for another hit later. This is worth knowing because 31% of customers covered in the report who suffered one attack saw another within the following 12 months!
The Internet of Things only makes the rise of attacks on companies more likely and more challenging to deal with as it continues to grow; more than 20 billion new devices are forecast to connect to the internet this year alone. Malware creators are ready and waiting to infiltrate the software underpinning these devices. This mounting threat landscape is something that developers cannot afford to ignore.
“The greatest ethical challenge facing software developers today is how to secure information in to reduce the risk of intellectual property loss to external threats (like hackers) in shared environments without affecting ease of use for the end user,” said Thomas Holt, a professor in the school of criminal justice at Michigan State University, whose research focuses on computer hacking and malware.
“If encryption or security becomes too cumbersome, employees will be less likely to use them or create workarounds, so there must be thought given as to how to protect vulnerable information in a distributed and networked environment.”
A Code of Ethics developed by the Association for Computing Machinery (ACM) states that computing professionals should “design and implement systems that are robustly and usably secure.” They should do this by integrating mitigation techniques and policies, such as monitoring, patching, and vulnerability reporting. It’s also important that developers take steps to ensure parties affected by data breaches are notified in a timely and clear manner.
The inspiration to consider the repercussions of a product team’s actions can come from any source:
“As a practicing security professional in tech, I’m grateful to see that many modern engineers were influenced by Isaac Asimov, inventor of the Three Laws of Robotics,” says Ty Sbano, Sisense Chief Security and Trust Officer. “Rule number one is ‘A robot may not injure a human being or, through inaction, allow a human being to come to harm.’ Even though the term ‘injure’ may be severe, if you take a step back, the mandate for product teams remains the same when it comes to acting ethically: In the case of privacy and ethics, or any other impact your creation could have on users and the world, you really need to think these things through.”
“Where machines learn like humans and from humans, unconscious bias is as much a threat as with humans.”
Lothar Determann
User acceptance testing and other best practices can help developers avoid implementing security precautions that are too confusing, are situationally inappropriate, or otherwise inhibit legitimate use. ACM is clear that there should be no compromise here: in cases where misuse or harm are predictable or unavoidable, it says the best option may be to not implement the system.
Give your customers (and everyone else) privacy
Software developers are often asked to create solutions that enable the collection, monitoring, and exchange of personal information. Our increasingly global workforce collaborates via a host of digital technologies spanning multiple countries and territorial borders. Employers can use many of these technologies to monitor employees, but at what point is it considered an invasion of privacy? How far do software developers integrate this ability to monitor into their solutions?
“Respect privacy” is featured high up in ACM’s Code of Ethics, which says that software developers should only use personal information “for legitimate ends, and without violating the rights of individuals and groups.” This means taking precautions to prevent the re-identification of anonymized data or unauthorized data collection, ensuring the accuracy of data, understanding the provenance of the data, and protecting it from unauthorized access and accidental disclosure. Personal information gathered for a specific purpose should not be used for other purposes without the person’s consent.
Fight algorithmic bias to deliver better products
Algorithmic bias describes systematic and repeatable errors in a computer system that create unfair outcomes. This danger looms greater than ever today thanks to machine learning techniques that are fundamentally changing the way software is made.
“Instead of coding instructions for machines from A-Z, engineers provide massive data sets to programs with high-level instructions to figure out solutions on their own,” said Lothar Determann, partner at multinational law firm Baker McKenzie, and professor of law at the Free University of Berlin. “Where machines learn like humans and from humans, unconscious bias is as much a threat as with humans.”
ACM’s Code of Ethics states to “be fair and take action not to discriminate.” It says that technologies and practices should be as inclusive and accessible as possible and software developers should take action to avoid creating systems or technologies that disenfranchise or oppress people. Failure to design for inclusiveness and accessibility may constitute unfair discrimination. Determann believes that success requires developers to code prohibitions into algorithms.
“We must not rely on machines learning on their own what is and isn’t prohibited,” he said. “We need to do our best to avoid replicating unconscious human biases by training machines with insufficient data (e.g. outdated history books), supervising teams (e.g. lacking diversity), and procedures (e.g. failing to flag for machines where data sets are known to be incomplete). We have to develop countermeasures to reduce the risk of replicating human unconscious bias in AI. Diverse teams and constant validation and questioning should be part of the solution.”
Think before you build
Software developers are the first, and last, lines of defense against the misuse of technology. Our current era of software culture may have been dominated by Facebook founder Mark Zuckerberg’s now-famous motto “move fast and break things,” but that’s not what users want from their software anymore. However, inside software companies, there can still be pressure on developers to get software to market quickly, making it tempting to skip rigorous testing.
“Software developers are often infatuated with the technology itself — what it can do and how we can apply it,” said Michael S. Kirkpatrick, associate professor in the department of computer science at James Madison University in Virginia, and education coordinator for ACM’s Committee on Professional Ethics.
Kirkpatrick argues that it’s the responsibility of developers to ask more questions about how their technology might be used. “All too often, developers are given a task to develop a piece of code, without information on the context in which it might be used. This needs to change. Developers need to be more proactive in finding out how the code is going to be used and anticipating how it might be misused. They should also question what policies and procedures can be put in place to prevent misuse.”
Kirkpatrick believes that multidisciplinary development teams are crucial to success here. “Software developers tend to have a very narrow focus on technology, so it’s really important that they work with ethicists, anthropologists, sociologists, and other people who bring a different perspective on the context of how a piece of technology might be used, and how misuse can be prevented.”
Developing a better world
It’s clear that a lot of responsibility rests in the hands of software developers. The ACM’s Code of Ethics is a useful framework for helping software engineers to live up to their ethical obligations, but it doesn’t provide a solution for solving ethical problems; rather it serves as the foundation of ethical decision-making.
Ultimately, it is up to software developers to act responsibly, taking the time to consider the wider impacts of their work, which should consistently support the public good.
“Technology amplifies power,” concludes Kirkpatrick. “Governments, large corporations, and other actors can use it to extend their influence in all aspects of our lives. It’s no exaggeration to say that the unquestioning adoption of technology poses risks to fundamental human freedoms and civic rights. So it’s really important for software developers to listen to different voices, to ask more questions and to think more proactively about how their software can be used and, more importantly, how it might be misused.”
Lindsay James is a journalist and writer with over 20 years’ experience creating compelling copy for some of the world’s biggest brands including Microsoft, Dassault Systemes, Exasol, and BAA. Her work has appeared in The Record (a magazine for the Microsoft partner community), Compass, and IT Pro.