Category Archives: Sisense
Sisense for Cloud Data Teams: a Step Forward for Builders Everywhere
Sisense News is your home for corporate announcements, new Sisense features, product innovation, and everything we roll out to empower our users to get the most out of their data.
A vision for builders
Nearly eight years ago, I joined my friend and Periscope Data co-founder Tom O’Neill (now Sisense Chief Cloud Officer) on a journey to build an advanced analytics platform that empowered data teams to build amazing new things. Last May, that journey took a big leap forward with the decision to merge Periscope Data and Sisense. In the time since that announcement, our team has been working extremely hard to combine technologies, customer bases, and internal operations into a single organization that achieves the goal we set out to accomplish in 2012.
Recently, that merger matured to a point where we felt it necessary to change the way we talk about the products. Internally, we were no longer keeping Periscope Data and Sisense separate products in our long-term vision and we wanted to make the external conversations we were having consistent. Rather than building two separate data analytics platforms for two separate parts of the overall data analysis process, we are focusing on building a single data platform with tools optimized for every user.
To reinforce that cohesion, we renamed Periscope Data as Sisense for Cloud Data Teams. When we began setting important checkpoints for the progress of this merger, this change was always an important milestone. I’m proud to be able to say that we’ve officially achieved it.
Building a shared culture
To this point, the merger has been a success story. Part of that success is due to our incredible customer base, which has been eager to listen to the shared vision and work with us to make it a reality. The truth is that building an ambitious product like this requires equally ambitious customers who can be relied on to take our data analytics platform to new heights and provide feedback along the way. It is neither a nicety nor an oversimplification to say that we could not have done this without the brilliant data professionals who use Sisense every day.
At this time last year, Periscope Data and Sisense both had strong company cultures. Merging them together required shared focus and developing a new culture around that commonality. For us, that common goal was a dedication to customers. Sisense for Cloud Data Teams is a testament to how well the new culture has jelled around servicing our combined customer base.
While the two original products focused on different users and unique parts of the data analysis process, the end-to-end platform available today is the result of a shared passion for empowering customers to create value from data. There’s a lot of enthusiasm about the data platform we are building and how it will be used to help out customers move their organizations forward.
Rewarding our internal builders
Another big pillar of the culture at Sisense is a commitment to employee success. Independently, both Sisense and Periscope Data had made several “best places to work” lists before the merger, which is illustrative of the priority given to employee satisfaction. There’s no pressure to produce perfect results, we’ve built an atmosphere that encourages continual experimentation and rewards those who help others. We set high standards for our work, give employees the resources they need to hit their goals, and trust that they’ll succeed.
That attitude is a huge reason why Sisense for Cloud Data Teams is available today and why future innovations at Sisense will set new industry standards for data platform possibilities. Renaming a product is not a simple process. It’s one that takes cooperation from all parts of the organization.
While the announcement of Sisense for Cloud Data Teams is a big reason to celebrate, there’s an understanding around the company that we’re not just building a product, we’re building a culture. The success of one goes hand-in-hand with the success of the other.
Building for the future
The announcement of Sisense for Cloud Data Teams is a milestone commemorating our dedication to building tools for advanced data professionals. It’s an industry-leading tool that already plays a crucial role in the way the most advanced companies are analyzing data. The success of that product so far is the result of individual people — both customers and employees — who share a vision for the possibilities of data and work to achieve it.
Sisense for Cloud Data Teams is also the next step forward in the journey to empower data teams. It removes existing workflow barriers between data professionals and the rest of their teams, streamlining the path from actions to insights. This tool is a leap ahead for modern data teams. The unified team Sisense and Persicope Data tea is building something truly incredible.
Harry Glaser is the Chief Marketing Officer and General Manager of San Francisco at Sisense. He was the co-founder and CEO of Periscope Data, which merged with Sisense in May 2019. Prior to founding Periscope Data in 2012, Harry was a Product Lead at Google AdWords and graduated from the University of Rochester with a bachelor’s degree in computer science.
Deriving Value from Data Lakes with AI
Artificial Intelligence and machine learning are the future of every industry, especially data and analytics. In Growing Up with AI, we help you keep up with all the ways this pioneering technology is changing the world.
Use AI to tackle huge datasets
Let’s talk about AI and machine learning (ML). Does your company actually have the tools and processes to use these innovations? Or do you just say that you have them? “AI-washing” is a disturbing trend among tech companies today, where they (correctly) ascertain that AI will be the deciding factor in the next generation of technologies, but don’t actually have the AI horsepower to back up their product, service, or claims. They slap a term like “AI-powered” or “AI-enhanced” to their offering and hope it’s enough to keep customers engaged.
“AI-washing” is a disturbing trend among tech companies today, where they slap AI language onto their product or service, but don’t actually have the AI horsepower to back up their claims.
However, half-measures just won’t cut it when it comes to handling huge datasets. Data is growing at a phenomenal rate and that’s not going to stop anytime soon. Many companies are already sitting on datasets so massive that it’s not possible for humans to deal with them unaided. AI and ML are the only ways to derive value from massive data lakes, cloud-native data warehouses, and other huge stores of information. This is both a daunting challenge and an inspiring opportunity, since effective use of AI and ML can cut through about 80% of data preparation (the annoying, routine stuff), leaving humans to handle the remaining 20%, the actual modeling and optimization.
Overcoming the obstacles between you and revenue
Once your data is prepared for analysis, the next question is: how else can AI help you? There’s a belief held by many in the AI and analytics worlds that the Holy Grail of data strategy is the ability to turn every question the business could ask (strategic, tactical, and operational) into requirements that an AI system can understand. Then the AI would go into the data, find the respective answers, and serve them up in easy-to-digest language that even a nontechnical team member can handle. This simple goal of “business questions to ML” would massively alter the business world, allowing users of any technical skill level to effortlessly draw deep insights out of their data. However, this sweeping paradigm shift will require the work of countless AI experts to tackle challenges that the industry simply hasn’t figured out yet.
There just aren’t enough AI and data science practitioners to go around to tackle this lofty goal. A recent Gartner report estimates that “by 2020, 50% of organizations will lack sufficient AI and data literacy skills to achieve business value.” Apply that metric to any other business-critical function. Can you imagine if 50% of companies didn’t have coders or sales reps or any of the other vital roles that keep the lights on? It’d be considered a crisis and an emergency. That’s the state of AI. Implementation of AI elements is expected to almost triple in the US while the overall supply of AI experts will stay significantly lower than demand.
Speaking of the demand, there’s another hurdle to AI success in the short-term. Business teams are looking for very specific tactical insights from their AI utilities (profitability, sustainability, brand, customers, etc.) compared to the Data Science/AI teams, which are more interested in strategic improvements including Natural Language Processing, computer vision, optical character recognition, and neural nets. Further complicating matters, research and development groups focus on bread-and-butter issues like scalability, security, reliability, and performance. Communication between these disparate teams, each with their own goals and desires, means that often novel AI solutions and applications are created and abandoned because stakeholders elsewhere in the company don’t understand the value they produce.
Get money out of your data
So where does this leave companies that want to survive and thrive in the coming AI apocalypse?
Most existing business models are somewhere in the “diminishing returns” region of the economic value curve (investing more resources while revenue stays flat). More and more funds are poured into R&D with no additional associated revenue coming in. The right data and analytics platform can help you bridge the gap between your current AI and analytics paradigm and where you want your company to be in the future.
Your platform should be able to effortlessly connect to numerous large datasets: cloud-native data warehouses and data lakes with millions or billions of rows. It should also be able to handle live datasets that constantly update and allow you to mash them up with cached data. If the system also allows for in-database preparation and materialized views that can be optimized for faster queries and lay the foundation for advanced analytics with Python and R, even better. All these utilities put powerful tools into the hands of data engineers to perform more complex analysis and use ML on huge datasets.
Whatever you want to do with AI and ML in the long term, the right analytics platform will be key to building it. It’s also a better way to monetize your data in the short term. It’s a win-win that more and more companies will rush to embrace as they undergo digital transformations and cloud migrations. It’s the future.
Inna Tokarev-Sela, Sisense’s Head of AI, has over 15 years’ experience in the tech industry. She spent the last decade at SAP, driving innovations in cloud architecture, in-memory products, and Machine Learning video analytics. A frequent speaker at industry events like IBC, NAB, WonderlandAI, and Media Festival, Inna holds a BS in Physics and Computer Science, an MBA, and an MS in Information Systems, having written her thesis on Neural Networks.
Periscope Data Expands to Israel, Empowering Data Teams with Powerful Tools
With data growing at a staggering rate, managing and structuring it is vital to your survival. In our Event Spotlight series, we cover the biggest industry events helping builders learn about the latest tech, trends, and people innovating in the space. In this piece, we detail the Israeli debut of Periscope Data.
Driving startup growth with the power of data
Growth. It’s the aspiration of every startup. The challenge is to do it right, and a crucial way to achieve it is with decisions based on data and analysis that drive measurable business results.
This was the key learning from the Sisense event heralding the launch of Periscope Data in Tel Aviv, Israel — the beating heart of the startup nation. We hosted over 150 people from more than 100 companies, who gathered to learn why data can supercharge their companies and how harnessing the huge power of data can take business from startup to unicorn.
An exciting slate of presentations took them on a journey from why to how they should use data analytics to optimize their operations successfully and maximize their business opportunities.
There are plenty of opportunities for startups, but huge challenges too, with more than 50 percent of US startups failing to last longer than four years. That’s why it has become increasingly vital for companies to leverage fresh data and deploy accurate analytics, so they can make smarter decisions that will enable them to prosper. Data teams have become the key to many organizations’ success, and the transformation of data into insights is now a critical business process, as Sisense Israel Sales Country Manager, Davidi Boyarksi, explained in his introduction.
Periscope Data provides the ideal platform for data teams to establish an effective analytics environment in the type of emerging companies that are so prevalent in Israel. It’s why Sisense, having merged with Periscope Data in May 2019, chose to host this event in Tel Aviv.
What VCs want from startups
The day’s first guest speaker, Itzik Parnafes, General Manager of global technology-focused VC investor Battery Ventures, set the scene, with an enlightening overview of the KPIs that VCs focus on during startup growth. He outlined how critical measurable results are to help VCs make major investment decisions — metrics such as revenue, net vs gross earnings, sales, costs and projections, and more. The importance of giving this information to VCs reinforces how significant a role data plays in measuring and predicting growth and enabling companies and investors to gain insights that drive growth.
The rise of the data team: from startup to unicorn
With Itzik’s wisdom fresh in everyone’s minds, Scott Castle, Sisense General Manager, Data Business, shared his view on the role of modern data teams. Scott whisked us through the history of business intelligence from its first definition in 1958 to the current rise of Big Data.
He talked through how the mind-blowing escalation of data and the drastic reduction in the cost of its storage has led to more complex, sophisticated uses of data and a shift in the way it’s managed and consumed. And he explained that the leap to visual-based data discovery put analytics in the hands of data experts working in different lines of business, rather than in IT, where bottlenecks could previously occur, and reporting could take much longer.
Scott outlined how this change has driven a shift in the role of data teams, who now occupy strategic business positions. These teams now play a far more significant part in shaping company strategy than ever before, because the opportunity of Big Data necessitated skilled data professionals not just to crunch the numbers but to find value in the information. Scott said:
Periscope Data’s mission is to turn data teams into super heroes because we see our role as unlocking the potential that data teams already have.
Scott Castle, Sisense General Manager, Data Business
And he emphasized why they should be treated differently. They’re no longer simply a repository for information. Instead they operate as an engine for transforming their organizations, using predictive technologies to answer business questions that shape decisions, roadmaps, and future strategies. He concluded that data teams can influence the transformation of startups into unicorns. A platform like Periscope Data is what makes this happen, by elevating data’s prominence and the role of data teams beyond merely a source of visual-based data discovery.
Responding to Scott’s presentation, attendees from AppsFlyer, Ziv Ben Naim, Client Services Analytics & Strategy Lead, and Maayan Dukas Kfir, Data Analyst had this to say:
It has been good to get aligned with what’s out there from a data team’s point of view. We really liked Scott’s story of the development of BI and Big Data and the vision he presented. It was interesting to get such a data orientated perspective, today, and as data specialists, it really resonated with us.
Ziv Ben Naim, Client Services Analytics & Strategy Lead, AppsFlyer
Maayan Dukas Kfir, Data Analyst, AppsFlyer
The impact on customers
The impact of the changes in the data ecosystem has already taken effect in numerous startup customers in Silicon Valley, and Kyle Dempsey, Senior Sales Engineer at Sisense shared some great examples of how Periscope Data has helped them grow from infancy to unicorn. Quoting Keystone Research, he opened with the finding that:
Companies who use data effectively have 18% higher gross margins and 4% higher operating margins
Keystone Research
And he demonstrated how the Periscope Data platform overcomes the challenges of huge data volumes that can’t be easily modeled by traditional BI. Citing Tinder as a major example, Kyle explained how it constantly uses data to enhance users’ interactions and calibrate the user experience. No surprise then that Tinder beat Netflix to become the highest-earning non-game app on both Google Play Store and the Apple Store.
Similarly, Kyle outlined how Flexport, the world’s first international freight forwarder and customs brokerage built around an online dashboard, uses Periscope Data to analyze billions of records, and get answers in seconds. Periscope Data enables Flexport to investigate critical business issues, identify opportunities and recommend ways to streamline business and operational processes by providing the company with the fastest time to insight, and powerful capabilities to answer any business questions, whether they derive from BI reporting, advanced analytics, or data prep for machine learning. Kyle said:
We empower data analysts to create more business value than any other BI platform.
Kyle Dempsey, Senior Sales Engineer, Sisense
These customer examples demonstrated how impactful smart data management and analytics can be for every part of customer organizations, from data teams to marketing, sales, and beyond. Another of Kyle’s case studies reinforced this point: Oscar Health, a disruptor in the private health insurance sector. The company has integrated data analysis throughout its organization to power decision making. From a startup in 2012, it is now valued at $ 3.2 billion. A true unicorn.
Optimizing data pipelines: How Kongregate uses Periscope Data
All these presentations provided rich food for thought, but there was more to follow. We were delighted to welcome Max Murphy, Software Engineering Manager for one of our customers, Kongregate, a leading mobile and PC game developer, publisher, and web gaming portal.
Kongregate has been using Periscope Data since 2013. Max demonstrated how Periscope Data enabled Kongregate to replace its ETL (extract, transform, load) system and apply software development best practices from end to end. The key element was standardizing its data pipeline from event source to charts, enabling Kongregate to gather, manage, and analyze a larger volume of data from across all data sources.
Additionally, Max discussed how embedded analytics offers huge benefits both to the company and third-party developers with whom they partner. He explained:
The big piece is embedding analytics in the Kongregate platform so game developers can see the performance of their games
Max Murphy, Software Engineering Manager, Kongregate.
The easy set-up and access to embedded analytics enable them to measure KPIs, get game statistics, monetization and retention statistics that help them to optimize players’ experience, hone best practices and benchmarks, and maximize stickiness and profitability.
Diving deeper into the datasphere: Data lakes — best practices
The day’s journey from the “why” through the “how” of data analytics then continued. We dived deeper into the mechanics of building the most efficient data pipelines to access and deliver data for analysis, even with the challenges of Big Data.
Omid Vahdaty, CTO of Jutomate Ltd., shone the spotlight on best practices with data lakes. As a specialist in Big Data solution architecture and systems engineering, Omid shared lessons from his extensive experience designing systems in startup environments. He addressed the challenges presented by today’s huge volume of data that’s enabled by cloud storage services. This data is often complex and difficult to model, particularly when it can be stored in both transformed and raw, structured, and unstructured formats. Omid discussed how to build data pipelines using data lake technologies, including the architecture and the guidelines needed to build your own data lake, using leading cloud vendors.
And extending this deep dive, Kyle stepped up once more to walk everyone through the ways Periscope Data handles data. His comprehensive demo showed how the SQL, R, and Python languages can be utilized and can work together to enable data teams to optimize performance.
This particular capability caught the attention of Dror Katz, Lead Data Analyst at Toluna (market research company), who was at the launch. He said:
We wanted to get familiar with new technology. We were familiar with Sisense before this event. It has been great to learn what capabilities Periscope Data has in comparison to other BI platforms. What stood out was the integration with Python and the added value that Periscope Data can offer.
Dror Katz, Lead Data Analyst at Toluna
And his views were reiterated by the AppsFlyer team attending the event, who learned about Periscope Data for the first time, and as a result, felt that:
. . . now we know more about what tools we’re missing, specifically regarding the integration of Python, SQL and R, which we don’t have with our current tools.
Ziv Ben Naim, Client Services Analytics & Strategy Lead, AppsFlyer and Maayan Dukas Kfir, Data Analyst, AppsFlyer
How Namogoo built a data-driven culture with Periscope Data
Of course, seeing how Periscope Data works in practice adds more color to the theory. Our final guest, Ohad Greenshpan, Co-Founder and CTO of Namogoo, presented a fascinating case study. Namogoo is a SaaS company that helps online businesses prevent customer journey hijacking. Its technology detects and blocks unauthorized ads injected into consumer browsers that redirect website visitors to competitor products and promotions.
He explained how Namogoo has used Periscope Data to grow its business and develop a data-driven culture at the company. Using the platform has empowered its teams to identify opportunities and take ownership of developing strategies informed by insights. Ohad explained that it has helped them discover things that they never even thought about, which has beneficially influenced decisions. He articulated the biggest benefit of all in one of his closing remarks:
The power of Periscope is in its simplicity.
Ohad Greenshpan, Co-Founder and CTO of Namogoo
Making an impression
Our launch of Periscope Data in Israel clearly left a positive impression. Dan Postar, CEO, Business Wise Solutions (Big Data and analytics consultancy), who already knows Sisense, said:
I think Periscope Data is a great addition to the Sisense capabilities. It was great to get an introduction to Periscope’s capabilities and an overview of what we can get from the Sisense suite.
Dan Postar, CEO, Business Wise Solutions
The event demonstrated that advanced BI and analytics are invaluable for startups facing intensified challenges from an environment that’s increasingly competitive and rich with data. Managing data so that organizations can more confidently make smart decisions is fast becoming a prerequisite for growth.
Periscope Data is perfectly designed to help dynamic startups and fast-growing companies achieve this. It adds to the armory of innovations that gives Sisense the capability to support organizations at every stage of growth, from startups to enterprise to OEM.
We’re excited to have launched Periscope Data into this fertile market, and we look forward to the opportunity to help accelerate the growth of many fantastic businesses.
Adam Murray’s early career PR started in London, then moved on to New York and Tel Aviv. He’s spent the last ten years working with tech companies like Amdocs, Gilat Satellite Systems, and Allot Communications. He holds a Ph.D. in English Literature and when he’s not spending time with his wife and son, he’s preoccupied with his beloved football team, Tottenham Hotspur.
6 Tips for Data Teams to Improve Collaboration
Everyone wants to get more out of their data, but how exactly to do that can leave you scratching your head. Our BI Best Practices demystify the analytics world and empower you with actionable how-to guidance.
In the right hands, data is the ultimate means to answer important business questions. The problem is that when data is used incorrectly, it still provides answers (just bad ones). The best way to avoid decisions made based on that bad information is to improve the relationships between the employees analyzing data and the employees acting on that information.
Today, data questions often involve someone from a line-of-business team and someone from a data team. Both people bring their own individual expertise to the collaboration, but these projects can easily fall into the trap of unclear expectations and insufficient communication.
Before building a dashboard to answer questions, data experts need to sit down with their line-of-business counterparts and have a discussion about the purpose of that dashboard. Here are a few tips for data experts to get the most out of that meeting:
Go into every dashboard with an open mind
It’s crucial to start every new data inquiry with a fresh mind. Every assumption the data experts make is an opportunity for the overall insight to lose value. On the other side, provided assumptions and business context can accelerate the data work for a faster time to value. The goal of every dashboard creator in a data request meeting should be to fully understand the data consumer’s needs and workflow. That means listening to their individual context and letting the investigation go where the data leads. If you drive the conversation or ask leading questions, you’ll end up at the resolution that you want, not necessarily the one that’s most valuable.
Get to know the individual requesting the data
Like most collaborative projects, empathy is critical to success. I like to start data request meetings by asking the data customer to walk me through their typical day. In some cases, it even helps me to shadow them for a while. What I’m looking for is a complete picture of the way that individual uses data. There might be pain points or missed opportunities for data to be used that I can help integrate. Asking someone about their typical day is also an easy way to get them communicating openly. It disarms them and puts the focus on the personal connection rather than a business problem.
Understand the business value behind a data request
An easy way for a data inquiry to get off track is for the dashboard creator to receive a request and cut straight to building a dashboard. Sometimes a request for a specific metric might miss the bigger question that data can solve. For example, a CS person might say “give me churn” but what they mean is “we need to find a way to minimize churn.” If you were to build a chart that simply listed churn over time, you’d miss all the other data points that correlate with churn. It’s often the data team’s job to connect data from different teams and maximize business value. Narrowing in on one team’s KPIs is an easy way to take your eye off the real goal. If you understand what is at the heart of the data request, it’ll be easier to work backward and find the right questions for the data to answer.
Work in pairs if possible
It’s true that two heads are generally better than one, but this concept is less about brainpower than it is about having one set of hands dedicated to note-taking. While a second data expert can definitely help understand the bigger questions in a meeting about data, it is invaluable to the flow of the conversation to have someone designated as a note-taker. Since the conversation moves at the speed of the slowest participant, it is extremely disruptive to have one person focusing on both moving the conversation forward and also documenting the important information for later. If a paired approach isn’t possible, it’s always an option to record the conversation, but I also recommend sketching out potential charts during these sessions, and that doesn’t translate well through audio.
Start general, then get specific
It’s likely that the initial dashboard request will be for a very pointed metric. This isn’t a bad way to start, but it’s the data expert’s job to guide the bring the focus to a bigger general question that this data can solve and uncover complementary data that can be included in that dashboard. Consider the example of churn from earlier. Presumably, the company’s goal is to maximize revenue and the inquiry into churn is being done to that end. It’s probably valuable to first create a variety of charts that illustrate the effect of churn on revenue. From there, it might also be useful to consider churn for individual cohorts, find factors that correlate in some way with churn or locate levers that will improve churn. Organizing the questions you ask from general to specific will also help you organize the charts on your dashboard when it’s time to create that asset.
Build an unsorted list of questions to answer with data
At the end of the meeting, the best thing to come away with is a list of questions that can be translated into data queries. An easy way to build this list is to open a document and just list the big questions that come up as you have exploratory conversations and try to understand the bigger business issue. If you can work as a pair, your note taker can do this part too. When the meeting is over (or maybe afterward if you recorded the conversation and want some time to go back and review notes), you can submit the list of questions to the dashboard requester and make sure that you have everything covered. Once that list is approved, you can start mapping each of the questions into a chart that will be organized and placed on your final dashboard.
Thinking long instead of short
Building a dashboard isn’t a straight line: there’s a lot of back and forth, questioning, editing, and iterating between the initial request and the finished product. These tips will help you build a strong foundation for this process and leave you in the best position for creating a dashboard that can really help your users make smarter decisions. Try embracing a collaborative process; you’ll be glad you took the time, asked the hard questions.
Christine Quan is a seasoned data and analytics veteran, focused on data visualization theory and building tools to empower data teams. She’s an expert at constructing SQL queries and building visualizations in R, Python, or Javascript.
Better Data Requests = Better Data Results
Building a Data-Driven Future
It’s important for everyone at a company to have the data they need to make decisions. However, if they just work with their data team to retrieve specific metrics, they are missing out. Data teams can provide a lot more insights at a faster rate, but you will need to know how to work with them to make sure that everyone is set up for success.
Data teams can be thought of as experts at finding answers in data, but it’s important to understand how they do that. In order to get the most value out of your collaboration, you need to help them understand the questions that matter to you and your team and why those questions need to be answered. There are a lot of assumptions that get built into any analysis, so the more the data team knows about what you are looking for, the more knowledge they may find as they explore data to produce their analysis. Here are four tips to make more productive requests from members of your data team:
Approach data with an open mind
It’s important to treat the data request process as an open-ended investigation, not a way to find data that proves a point. A lot of unexpected insights can be found along the way. Make your goal to ask questions and let your data team search for the answers. This approach will allow you to get the best insights, the type of unknowns that could change your decision for the better. If you put limitations on what you’re asking the data, you’ll end up putting limitations on the insights you can get out of your inquiry.
To really dig into this, think about how questions are answered scientifically. Scientists treat any bias as an opportunity for the insight to be compromised. For example, let’s say you are looking to improve customer satisfaction with your product. Requesting a list of customers with the highest and lowest NPS scores will give you a list of people who are happiest or most frustrated, but it is not going to let you know how to improve customer satisfaction. This request puts too much attention on the outliers in your customer base rather than identifying the key pain points. That’s part of the picture, but not all of it. If you’re trying to create a program that targets your goal, let your data team know the goal, give them a few optional starting points, and see what they come back with. They might surprise you with some sophisticated analysis that provides more insight and helps you launch a fantastic program.
Start with a conversation, not a checklist
The single biggest mistake a line-of-business professional can make when requesting data is to present a data expert with a list of KPIs and tell the data team to just fill in the blanks. This approach misses so much of the value a data team can provide. Modern data teams have technology and abilities that allow them to go much further than just calculating numbers. They can guide analytical exploration through flexible, powerful tools to make sure you’re getting the most valuable insights out of your data.
Instead of a list of metrics, think about starting your data request as a meeting. You can provide the business context needed and a list of questions that you want answered. You can even present some initial hypotheses about what those numbers may look like and why they might move in one direction or another. This is a great way to kick off the conversation with your data counterpart. From there, you can benefit from their experience with data to start generating new and more informed questions from their initial inquiries. The data team’s job is to get you information that helps you be more informed, so give them as much context as possible and let them work as a problem solver to find data-driven recommendations.
Data should recommend actions, not just build KPIs reports
A lot of standard business KPIs measure the results of company efforts: revenue, lead conversion, user count, NPS, etc. These are important statistics to measure, but the people tracking them should be very clear that these numbers track how the company is moving, not why it is moving that way. To make these data points actionable, you need to take analysis further. Knowing that your NPS is going up or down is useless if it doesn’t inform a customer team about the next step to take.
A good data team will map important KPIs to other data and find connections. They’ll comb through information to find the levers that are impacting those important KPIs the most, then make recommendations about how to achieve your goals. When you get a list of levers, make sure to understand the assumptions behind the recommendations and then take the right actions. You can always go back to those KPI reports to test if the levers are having the intended effect.
Data requests are iterative, give the data person feedback
Communication about data should not end when the data has been delivered to you. It’s important to dig into the analysis and see what you can learn. Instead of reporting that data or taking action on it right away, you should check with your dashboard creator to make sure that he or she can verify that you’re reading all of the data properly and that the next steps are clear. There are a lot of ways to misinterpret data; a good way to prevent mistakes is to continue communicating.
Even if you’ve gotten the right takeaways from the data, it’s still good to consult with your dashboard creator and go over your interpretation of the information so they know how you read data. You may need a follow-up meeting to restart with the overall question you want to answer, then see what additional data needs to be collected or what modifications are needed to make the report or dashboard work best for your intended use-case.
Sisense at AWS re:Invent — Helping Enterprises Win in the Analytic Apps Economy
AWS re:Invent is the single most important gathering of builders every year. As an AWS Data & Analytics Competency partner, we’re honored to join and power this innovative community.
It’s an innovative community because it’s being fueled by a thriving economy — the analytic apps economy. Executives, business leaders, and Chief Data Officers are leading the way when it comes to generating new value and new revenue for their customers with analytics products and services. Companies that aren’t taking advantage of this trend will be left behind.
If you want to unlock new value for your customers and generate millions of incremental revenue leveraging Sisense and AWS, we’d love to help. Our executive leadership team will be at re:Invent to share their experiences. Register here to set up a time to meet!
A proven track record
If you’re not attending AWS re:Invent, but your company is on a journey to generate new value and new revenue for your customers, you’re in good company:
For instance, the leaders at GE Healthcare were early believers in the idea that they could drive more financial, operational, and clinical outcomes for their partners and customers by extracting actionable insights from the data they created and managed for them. Leveraging Sisense, they created millions in recurring revenue and added a strong market differentiator.
At Sisense, we work across industries on many of those opportunities. It’s now become clear that:
- All companies can participate in the multi-billion analytic and AI apps economy, creating new go-to-market and revenue streams
- Sisense truly is the best platform to support those innovations at scale, leveraging existing infrastructures, people, domain expertise, and markets
Maximizing return on existing investments
As an enterprise executive, it’s important to realize early in your career that a core path to success is not always the “rip and replace” or “different must be better” approach. It is far more beneficial to a company to find ways to accelerate what we call return on existing investment — the ability to create revenue from existing assets like data, market penetration, domain expertise, technology decisions, people, and operational excellence. Building a strategy to monetize incremental value from those assets is key to success in existing or new markets.
In fact, large enterprises are the only ones capable of innovation at scale. These enterprises are perfectly situated to successfully compete with the startup ecosystem, leveraging the following strengths:
That’s exactly what successful developers and product managers building apps on AWS and Sisense do: they select best-in-class technologies, design a flexible platform architecture, enable business lines to focus on delivering incremental value to their customers, differentiate through outcome selling, and experiment cost-effectively.
Sisense works with Fortune 500 companies across industries, unveiling opportunities that add up to millions of incremental, high-margin revenues, delivering five times that value to their customers. Every day our strategy consulting business and customer success teams help identify opportunities for millions in new revenue and advise businesses about how to organize, budget, and go to market. Some have built-in advantages: those with a solid AWS deployment with centralized data, fast time to build AI algorithms, and a developer-friendly interface to manage workflow applications have a head start in leveraging Sisense to embed actionable insights into their core products and monetize those assets.
Building vs. buying the platform
When technology was less mature, it might have made sense for enterprises to build and own both the platform and the apps. But that leads to more resources spent on technology and fewer resources invested in using your domain expertise to solve customer problems and deliver outcomes. That’s the opposite of what a smart business wants to do!
Companies can avoid that tradeoff by using a microservice-oriented architecture to create that new revenue. This year, the dream came true when in August we released the most modern native-cloud architecture available for builders. This technology is exactly what any company needs to significantly increase its AWS ROI (it was built for Amazon Linux 2). We worked closely with AWS on the product and continue to do so as we roll it out to customers.
If you’re not a platform company, you must understand the “build vs buy” debate: Competing on quantifiable outcomes is a much better use of existing internal resources (vs competing to develop the best platform) and customer time. Once you’ve set up the right AWS/Sisense infrastructure, all you have to do is to find what high-value questions need to be answered. Consider what types of new value you can create for your customers, your suppliers, and your end-users. Then deliver on that value and get a share of that outcome.
Meet you at re:Invent
Sisense is the prime way to supercharge your Amazon Redshift and overall AWS investments in today’s analytic apps economy. Let’s talk more at re:Invent! Register here to meet with the Sisense executive team or visit us directly at booth #837.
How To Calculate Average Sales
No matter what industry you’re in, any sector that deals with customers will have to keep track of their sales. When you need a quick way to monitor your company’s success in meeting objectives, sales provide one of the easiest metrics since it is a direct display of efficiency related to profits. Even so, raw sales data can be overwhelming and may not always paint the clearest picture.
Using average sales across different periods can give you a better idea of how well your sales strategies and marketing campaigns are performing, which tactics are connecting with consumers, and how successful your sales team is at converting leads. More importantly, it gives you a straightforward way to establish a standard for measuring success and failure. Calculating average sales is an uncomplicated process and can help steer your business decisions for greater success.
Why measure average sales?
More than just an eagle’s eye view of your sales operations, average sales can also give you a granular view at the results of every sale. Measuring average sales by customer can deliver useful insights such as how many dollars customers are spending at the point of sale, and how that data compares to historical data.
On a broader level, you can compare the efficiency of different teams, stores, and branches by measuring their monthly and daily sales against historic averages and each other. This is important when choosing how to allocate budgets, deciding where to trim resources, and providing greater support. By understanding the historic patterns and combining them with more real-time data, you can make smarter decisions regarding your sales pipeline.
How to calculate average sales
Calculating your average sales depends on two factors: a period or frequency you want to analyze and the total sales value for that period. Average sales can be measured on a much smaller scale, such as daily or weekly, or on a larger scale like monthly and even annually.
To calculate the average sales over your chosen period, you can simply find the total value of all sales orders in the chosen timeframe and divide by the intervals. For example, you can calculate average sales per month by taking the value of sales over a year and dividing by 12 (the number of months in the year). If the total sales for the year were $ 1,000,000, monthly sales would be calculated as follows:
Average sales per month, in this case, would be roughly $ 83,000.
Daily average sales are also a common calculation, and they can vary based on the broader timeframe being measured. For example, you could measure daily average sales over a period of a single month to compare year-over-year data or calculate daily average sales over a full year to see how stores and sales teams performed throughout a 12-month period. In this case, the calculation would not change, except for replacing the top number for annual total sales, and dividing by the total number of workdays.
A variant average sales calculation
Another useful way to track the average value of a sale is to measure how effective your sales team is on a per-customer basis. While overall visitors and the number of sales may be on the rise, if the value of sales per customer is declining, your overall revenues may actually fall.
In this case, the division is similar to average sales, but instead of a time frame, you can divide the total sales value by the number of transactions completed during the period you are analyzing. For instance, if your total sales for the day were $ 15,000, and you completed 35 unique transactions, the average value of sales would be approximately $ 528 per customer. The formula to calculate average sales value is as follows:
Other KPIs you can include
Average sales are a great place to start tracking your sales effort, but to gain more actionable insights, your dashboard should also include other KPIs that can provide useful context. These are just a few of the useful sales dashboard examples of KPIs you can include when building your BI platform.
- Average Revenue Per Unit (ARPU) — This metric is like average sale value but measures how much revenue a single customer or user will generate. This number is found by measuring revenue against the total number of units.
- Sales per Rep — Average sales don’t give you a look into how individual salespeople may be performing. Adding sales per rep will provide a more granular look at your sales operations.
- Opex to Sales — Raw sales data provides insight, but little context. Understanding how operating expenses relate to sales helps clarify the real value of a sale. If the Opex is too high, even large sales offer little real value.
Looking for other ways to measure your sales numbers? Explore our interactive sales dashboards!
Cybersecurity Starts with You: Celebrating National Cybersecurity Awareness Month
Ensuring safety and privacy in today’s online environment is an enormous task. It’s way too much for a single person or a single company to manage on their own. As a security professional, I am personally encouraged by continual reminders that I’m not fighting this battle alone.
Every October is National Cybersecurity Awareness Month (NCSAM), and I’m proud to announce that Sisense has taken the initiative to sign up as a Champion for the cause. NCSAM is a great reminder that ensuring secure environments for online activity is a collaborative effort. It takes all of us — businesses, government agencies, colleges and universities, associations, nonprofit organizations, and individuals — working together to achieve the ultimate security goal.
This year’s NCSAM theme is “Own It. Secure IT. Protect IT.” The goal is to encourage every consumer to understand their everyday technology and prevent cybercrimes by securing their digital profiles. The NCSAM website has lots of information about dealing with more sophisticated cybercriminals and a list of standard tips to secure yourself online. Those best practices are a simple way for any individual or group to be cyber smart.
This is an exciting year for me to be a part of the NCSAM activities. Last year, I led the NCSAM efforts at Periscope Data, which were very successful. But this year, I get to oversee efforts to ingrain the security mindset in the DNA of a newly merged organization. As a Champion of the NCSAM efforts, it’s a huge opportunity for Sisense to step up and lead the charge at a much larger scale.
Now in its 16th year, NCSAM continues to build momentum and impact with the ultimate goal of providing all Americans with the information they need to stay safer and more secure online. Sisense is proud to support this far-reaching online safety awareness and education initiative which is co-led by the National Cyber Security Alliance (NCSA) and the Cybersecurity and Infrastructure Agency (CISA) of the U.S. Department of Homeland Security.
Keeping businesses secure
To be totally secure, individual employees need to do more than just rely on their company’s IT and security team to protect their devices and environment. I encourage any security professional to point their teams to the personal security tips on the NCSAM website.
As far as business cybersecurity is concerned, there are also tips for identifying devices, protecting data, responding to threats, and recovering breached information if an incident occurs. Here’s a quick overview of some of those pointers:
Have a recovery plan, even if you don’t need it
It’s easy to get enthusiastic about your security efforts and think that no attack could ever happen to your company’s data. But every company is always vulnerable, you need to prepare for the worst. That means familiarizing yourself with the breach notification laws in your state, preparing for an emergency workflow without certain technologies (even switching some operations to paper if necessary), and preparing spares/backups of certain data assets.
Stay up-to-date on the technology in your company
A good place to start this task is with an inventory of data and physical devices that the company owns. This list must be updated extremely frequently with information about where that technology is kept and who has access to it. A strong authentication process for employee accounts also helps manage access. Enforcing strong passwords can seem like a hassle, but it’s necessary in today’s challenging security environment. Cybersecurity professionals need to put forth strong messaging and help build a culture of security to drive home how vital it is that teams ensure that passwords aren’t an opportunity for a breach.
Know how your data is connected
Whether it’s individual devices or cloud pipelines, security professionals need to the people and devices that are connected to their data. Mapping out the flow of information is essential to every step of the security process. Regularly auditing your information connectors (and making sure that those connectors meet certain security standards) is a great way to keep business data safe.
When in doubt, throw it out
It’s a security professional’s job to train the rest of their organization about the threats that can be found in suspicious emails, posts, ads, messages and attachment. They should be taught a healthy suspicion of unfamiliar or unexpected activity. More than just being on the lookout for those potential attacks, teams need to know to report suspicious activities, emails, and messages so the security team can guard against future attacks. It’s an ongoing effort to stay on top of the most common threats and protect against them.
A deeper discussion of security
Earlier this month, I had a chance to discuss personal experiences from my career in security on Business Security Weekly. That conversation is full of individual tips to improve your own security and comments about the larger security industry as a whole. You can watch the entire episode of Business Security Weekly below.
For more information about NCSAM 2019 and a wide variety of personal and business security activities, visit staysafeonline.org/ncsam.
Data Engineering Today: All About the Cloud
When “data engineer” first started becoming a vital role for tech companies, the world was a smaller, simpler place. Engineers were primarily concerned with handling data stored in Excel spreadsheets and on local machines. New tools were slowly introduced to deal with data as it grew in volume and complexity, but the overall landscape was largely unchanged until the rise of “The Cloud.” Once Amazon Web Services and its cloud-storage-as-a-service model hit the tech world, what it meant to be a data engineer changed forever. Now all tech and all data is all about the cloud. And that’s a good thing! Today I’m going to talk about how the cloud changed everything and what else this technology will bring to the table.
Cloud changed everything
It can’t be overstated: the rise of cloud storage and computing changed everything for companies and data engineers alike, and it’s never going back. Engineers were always most concerned with connecting data services to analytics and business intelligence software or whatever other systems needed to use that data. As the first builders in a company’s data world, they assembled the data pipeline and kept it in shape.
However, with AWS and cloud solutions for everything, one of the main concerns that data engineers have today is keeping up with trends in these solutions. And because Amazon was first on the scene with AWS (and still has a huge market share), that means that engineers often need to be on top of what Amazon is up to. Certifications and training programs from Amazon and third-party providers are an essential part of being a skilled, competitive engineer.
Every company a cloud data company
Today, being a data engineer means connecting your company’s business systems to cloud-based data sources. This can be one cloud-native data warehouse where you dump all your information and sort through it later, or you could be dealing with a number of different cloud-hosted datasets from different departments that all need to be brought together into a centralized analytics platform or other software system. Cloud storage and processing is so cheap and convenient that every startup turns to the cloud for their computing needs. They’d be foolish not to. Cloud’s flexible pricing options make putting your data (and your whole app) on the cloud the smart decision.
If you are starting a company today, you’ll probably do everything on the cloud: data storage, code hosting, everything. You’ll never own a server machine again. That’s the past. Everything can be done on the cloud, so you can just worry about building the best application you can (and marketing it, scaling it, etc.) and let AWS (or whichever cloud-native data warehouse you choose) worry about the hardware, maintenance, uptime, etc.
Cloud is great for startups because as the company scales users and their need for data storage and processing power goes up, they can just pay a little more to keep using more resources on the other end. In fact, with the free starting tier that a lot of cloud platforms offer, a startup can stay on the free tier until they start making the money they need to pay for more resources. Meanwhile, if they take a dip in user numbers, they’re not paying for resources they don’t need. This is all great for the data engineer because once the cloud data pipeline is all set up, they don’t need to constantly tinker with it. Usage may fluctuate, but the pipeline perseveres.
Cloud is also the choice for enterprise-grade companies. If you’re an enterprise and you’re not already doing everything on the cloud, then you’re likely in the midst of a massive digital transformation. Storing data in the cloud just makes sense: no machines to maintain, easy access from anywhere, and easy integration with cloud-native architecture like Linux. The bigger and more complex the organization, the more important it is that their cloud datasets can easily connect with their application software. Linux is the OS of choice for situations like these because it allows for continual, scalable rollouts of new code and provides high reliability and availability. Enterprises either already love the cloud or are learning to love it, and the new generation of startups will likely never know any other place to store their data.
Another reason every organization cares about making its data accessible is because every company is now a data company. Sure, bringing data into an application for the business’s core function is vital, but it’s only one function that data performs in modern tech companies. More and more companies are realizing that data can be monetized for other purposes or mashed up with other sources and sold back to the users as insights in ways that add more and more value to their users’ lives. And doing this all on the cloud makes the whole process simpler and more efficient, whatever you’re doing with it. In-house analytics consumers, especially business users (who tend to be mostly nontechnical), will appreciate a speedy, well-maintained cloud data pipeline because it gives them the flexibility to answer all their questions, while serving up insights in a speedy manner.
The cloud horizon
Everything is moving to the cloud. Again: if you are starting a technology business today, you want to use cloud-native architecture and host it all on the cloud. Serverless solutions are the present and the future. No more worrying about hardware, just push your code and go. Data engineers will always be a vital part of making sure that data is going where it needs to go safely and securely, and staying on top of the latest cloud trends, even if the days of managing hardware and on-site datasets are mostly over. Check back for future blog posts where I’ll discuss why serverless cloud functions are so awesome, among other cloud computing topics.
REST API Testing Strategy: What Exactly Should You Test?
The API layer of any application is one of the most crucial software components. It is the channel which connects client to server (or one microservice to another), drives business processes, and provides the services which give value to users.
A customer-facing public API that is exposed to end-users becomes a product in itself. If it breaks, it puts at risk, not just a single application, but an entire chain of business processes built around it.
Mike Cohn’s famous Test Pyramid places API tests at the service level (integration), which suggests that around 20% or more of all of our tests should focus on APIs (the exact percentage is less important and varies based on our needs).
Once we have a solid foundation of unit tests which cover individual functions, API tests provide higher reliability covering an interface closer to the user, yet without the brittleness of UI tests.
API tests are fast, give high ROI, and simplify the validation of business logic, security, compliance, and other aspects of the application. In cases where the API is a public one, providing end-users programmatic access to our application or services, API tests effectively become end-to-end tests and should cover a complete user story.
So the importance of API testing is obvious. Several methods and resources help with HOW to test APIs — manual testing, automated testing, test environments, tools, libraries, and frameworks. However, regardless of what you will use — Postman, supertest, pytest, JMeter, mocha, Jasmine, RestAssured, or any other tools of the trade — before coming up with any test method you need to determine what to test…
API test strategy
The test strategy is the high-level description of the test requirements from which a detailed test plan can later be derived, specifying individual test scenarios and test cases. Our first concern is functional testing — ensuring that the API functions correctly.
The main objectives in functional testing of the API are:
- to ensure that the implementation is working correctly as expected — no bugs!
- to ensure that the implementation is working as specified according to the requirements specification (which later on becomes our API documentation).
- to prevent regressions between code merges and releases.
API as a contract — first, check the spec!
An API is essentially a contract between the client and the server or between two applications. Before any implementation test can begin, it is important to make sure that the contract is correct. That can be done first by inspecting the spec (or the service contract itself, for example a Swagger interface or OpenAPI reference) and making sure that endpoints are correctly named, that resources and their types correctly reflect the object model, that there is no missing functionality or duplicate functionality, and that relationships between resources are reflected in the API correctly.
The guidelines above are applicable to any API, but for simplicity, in this post, we assume the most widely used Web API architecture — REST over HTTP. If your API is designed as a truly RESTful API, it is important to check that the REST contract is a valid one, including all HTTP REST semantics, conventions, and principles (here, here, and here).
If this is a customer-facing public API, this might be your last chance to ensure that all contract requirements are met, because once the API is published and in use, any changes you make might break customers’ code.
(Sure, you can publish a new version of the API someday (e.g., /api/v2/), but even then backward compatibility might still be a requirement).
So, what aspects of the API should we test?
Now that we have validated the API contract, we are ready to think of what to test. Whether you’re thinking of test automation or manual testing, our functional test cases have the same test actions, are part of wider test scenario categories, and belong to three kinds of test flows.
API test actions
Each test is comprised of test actions. These are the individual actions a test needs to take per API test flow. For each API request, the test would need to take the following actions:
1. Verify correct HTTP status code. For example, creating a resource should return 201 CREATED and unpermitted requests should return 403 FORBIDDEN, etc.
2. Verify response payload. Check valid JSON body and correct field names, types, and values — including in error responses.
3. Verify response headers. HTTP server headers have implications on both security and performance.
4. Verify correct application state. This is optional and applies mainly to manual testing, or when a UI or another interface can be easily inspected.
5. Verify basic performance sanity. If an operation was completed successfully but took an unreasonable amount of time, the test fails.
Test scenario categories
Our test cases fall into the following general test scenario groups:
- Basic positive tests (happy paths)
- Extended positive testing with optional parameters
- Negative testing with valid input
- Negative testing with invalid input
- Destructive testing
- Security, authorization, and permission tests (which are out of the scope of this post)
Happy path tests check basic functionality and the acceptance criteria of the API. We later extend positive tests to include optional parameters and extra functionality. The next group of tests is negative testing where we expect the application to gracefully handle problem scenarios with both valid user input (for example, trying to add an existing username) and invalid user input (trying to add a username which is null). Destructive testing is a deeper form of negative testing where we intentionally attempt to break the API to check its robustness (for example, sending a huge payload body in an attempt to overflow the system).
Test flows
Let’s distinguish between three kinds of test flows which comprise our test plan:
- Testing requests in isolation – Executing a single API request and checking the response accordingly. Such basic tests are the minimal building blocks we should start with, and there’s no reason to continue testing if these tests fail.
- Multi-step workflow with several requests – Testing a series of requests which are common user actions, since some requests can rely on other ones. For example, we execute a POST request that creates a resource and returns an auto-generated identifier in its response. We then use this identifier to check if this resource is present in the list of elements received by a GET request. Then we use a PATCH endpoint to update new data, and we again invoke a GET request to validate the new data. Finally, we DELETE that resource and use GET again to verify it no longer exists.
- Combined API and web UI tests – This is mostly relevant to manual testing, where we want to ensure data integrity and consistency between the UI and API.
We execute requests via the API and verify the actions through the web app UI and vice versa. The purpose of these integrity test flows is to ensure that although the resources are affected via different mechanisms the system still maintains expected integrity and consistent flow.
An API example and a test matrix
We can now express everything as a matrix that can be used to write a detailed test plan (for test automation or manual tests).
Let’s assume a subset of our API is the /users endpoint, which includes the following API calls:
| GET /users | List all users |
| GET /users?name={username} | Get user by username |
| GET /users/{id} | Get user by ID |
| GET /users/{id}/configurations | Get all configurations for user |
| POST /users/{id}/configurations | Create a new configuration for user |
| DELETE /users/{id}/configurations/{id} | Delete configuration for user |
| PATCH /users/{id}/configuration/{id} | Update configuration for user |
Where {id} is a UUID, and all GET endpoints allow optional query parameters filter, sort, skip and limit for filtering, sorting, and pagination.
| # | Test Scenario Category | Test Action Category | Test Action Description |
| 1 | Basic positive tests (happy paths) | ||
| Execute API call with valid required parameters | Validate status code: |
1. All requests should return 2XX HTTP status code
2. Returned status code is according to spec: |
|
| Validate payload: |
1. Response is a well-formed JSON object
2. Response structure is according to data model (schema validation: field names and field types are as expected, including nested objects; field values are as expected; non-nullable fields are not null, etc.) |
||
| Validate state: |
1. For GET requests, verify there is NO STATE CHANGE in the system (idempotence)
2. For POST, DELETE, PATCH, PUT operations |
||
| Validate headers: |
Verify that HTTP headers are as expected, including content-type, connection, cache-control, expires, access-control-allow-origin, keep-alive, HSTS and other standard header fields – according to spec. Verify that information is NOT leaked via headers (e.g. X-Powered-By header is not sent to user). |
||
| Performance sanity: | Response is received in a timely manner (within reasonable expected time) – as defined in the test plan. | ||
| 2 | Positive + optional parameters | ||
| Execute API call with valid required parameters AND valid optional parameters
Run same tests as in #1, this time including the endpoint’s optional parameters (e.g., filter, sort, limit, skip, etc.) |
|||
| Validate status code: |
As in #1 | ||
| Validate payload: |
Verify response structure and content as in #1.
In addition, check the following parameters: Check combinations of all optional fields (fields + sort + limit + skip) and verify expected response. |
||
| Validate state: |
As in #1 | ||
| Validate headers: |
As in #1 | ||
| Performance sanity: | As in #1 | ||
| 3 | Negative testing – valid input | ||
| Execute API calls with valid input that attempts illegal operations. i.e.:
– Attempting to create a resource with a name that already exists (e.g., user configuration with the same name) – Attempting to delete a resource that doesn’t – Attempting to update a resource with illegal valid data (e.g., rename a configuration to an existing name) – Attempting illegal operation (e.g., delete a user configuration without permission.) And so forth. |
|||
| Validate status code: |
1. Verify that an erroneous HTTP status code is sent (NOT 2XX)
2. Verify that the HTTP status code is in accordance with error case as defined in spec |
||
| Validate payload: |
1. Verify that error response is received
2. Verify that error format is according to spec. e.g., error is a valid JSON object or a plain string (as defined in spec) 3. Verify that there is a clear, descriptive error message/description field 4. Verify error description is correct for this error case and in accordance with spec |
||
| Validate headers: |
As in #1 | ||
| Performance sanity: | Ensure error is received in a timely manner (within reasonable expected time) | ||
| 4 | Negative testing – invalid input | ||
| Execute API calls with invalid input, e.g.:
– Missing or invalid authorization token And so on. |
|||
| Validate status code: |
As in #1 | ||
| Validate payload: |
As in #1 | ||
| Validate headers: |
As in #1 | ||
| Performance sanity: | As in #1 | ||
| 5 | Destructive testing | ||
| Intentionally attempt to fail the API to check its robustness: Malformed content in request Wrong content-type in payload Content with wrong structure Overflow parameter values. E.g.: – Attempt to GET a user with invalid UUID – Overflow payload – huge JSON in request body Boundary value testing Empty payloads Empty sub-objects in payload Illegal characters in parameters or payload Using incorrect HTTP headers (e.g. Content-Type) Small concurrency tests – concurrent API calls that write to the same resources (DELETE + PATCH, etc.) Other exploratory testing |
|||
| Validate status code: |
As in #3. API should fail gracefully. | ||
| Validate payload:
Validate headers: |
As in #3. API should fail gracefully. As in #3. API should fail gracefully. | ||
| Performance sanity: |
As in #3. API should fail gracefully. |
Test cases derived from the table above should cover different test flows according to our needs, resources, and priorities.
Following the test matrix above should generate enough test cases to keep us busy for a while and provide good functional coverage of the API. Passing all functional tests implies a good level of maturity for an API, but it is not enough to ensure high quality and reliability of the API.
In the next post in this series we will cover the following non-functional test approaches which are essential for API quality:
Security and Authorization
- Check that the API is designed according to correct security principles: deny-by-default, fail securely, least privilege principle, reject all illegal inputs, etc.
- Positive: ensure API responds to correct authorization via all agreed auth methods – Bearer token, cookies, digest, etc. – as defined in spec
- Negative: ensure API refuses all unauthorized calls
- Role Permissions: ensure that specific endpoints are exposed to user based on role. API should refuse calls to endpoints which are not permitted for user’s role
- Protocol: check HTTP/HTTPS according to spec
- Data leaks: ensure that internal data representations that are desired to stay internal do not leak outside to the public API in the response payloads
- Rate limiting, throttling, and access control policies
Performance
- Check API response time, latency, TTFB/TTLB in various scenarios (in isolation and under load)
Load Tests (positive), Stress Tests (negative)
- Find capacity limit points and ensure the system performs as expected under load, and fails gracefully under stress
Usability Tests
- For public APIs: a manual “Product”-level test going through the entire developer journey from documentation, login, authentication, code examples, etc. to ensure the usability of the API for users without prior knowledge of our system.
Thanks for reading. To be continued!