Business Intelligence Info - What's happening in the BI World

How Sales Reps Can Solve 3 Nagging Document Problems

July 20, 2019 CRM News and Info

Let’s face it. The sales landscape has changed. Technology has enabled and empowered today’s customers and as a result, they are totally in control of the sales process. This means that the quality of the customer experience is a determining factor in whether a sales organization succeeds or fails.

Consider, for instance, that more than half of customers (54 percent) are likely to switch brands if a company doesn’t make an effort to personalize communications with them, and half of customers will consider switching brands if a company doesn’t anticipate their needs.

Today’s connected and educated customers also expect salespeople to be available at any time to provide immediate, direct and tailored responses. Spending too much time on activities other than selling is no longer an option for sales teams. They must be enabled with tools and processes that maximize efficiency and productivity, and focus on customers and driving revenue, or they risk falling behind.

Time Spent Not Selling

It’s no secret that salespeople spend a lot of their time doing activities outside of selling. In fact, close to just 36 percent of sales reps’ time is focused on the primary sales function, according to CSO Insights.

A large number of hours instead are spent on non-revenue-generating activities, like manual and administrative tasks, and many of them stem from the processes required to generate documents.

Consider the following:

The average sales rep spends more than 50 full days away from core selling activities each year (Domo).

For sales teams that see ineffective internal processes as their top challenge, 22 percent identified lack of time for face-to-face selling, and 45 percent identified excessive administrative tasks as the main reasons (Salesforce).

For sales teams to become high functioning and more successful, and ultimately focus more on providing the customer-centric experience required to meet customers’ needs, less-impactful activities need to be minimized.

The reality is that a good part of this manual work comes from the document-generation process, as salespeople must produce and send numerous documents throughout the sales cycle. It’s inevitable that if documents are created manually, there likely will be problems. Following are a few common problems that result from the manual document generation process:

1. Inconsistent document formatting and branding:

2. Poor visibility into document status: Documents that aren’t stored in a central repository and can’t be tracked can lead to compromised visibility. Imagine this scenario: An individual sales rep keeps what he thinks is the latest price book on his laptop and uses that information to generate quotes manually. Then he drops his laptop while running for an Uber, leaves the company, or goes on vacation for a few days.

During that time, a question from a customer comes up regarding a recent proposal. Was a proposal sent one week ago? Or was it two weeks? Did it contain the new pricing structure? Sales teams often need to answer questions like these quickly, and finding the answer can be challenging. Searching through sent emails and local computer drives can be challenging, time-consuming and unreliable when a central repository doesn’t exist.

3. Copying and pasting: While it may seem obvious, copying and pasting to create document content wastes valuable employee time and introduces errors. Most often this is done when there is an existing document template or an old version of a document that sales is copying and pasting into to create a new document.

This process is slow and laborious, and any time a sales person creates a document manually or transfers information into a template by hand, there is a chance for error. Document formatting can be lost or altered, text/characters might be omitted, or peripheral text might be missed. Errors can cause delays in the sales process, which is frustrating for both sales and prospective customers.

To help solve all of these problems, sales teams need a process that automatically can produce a document — whether it’s a Word document, Excel spreadsheet, HTML email or other type. In turn, document generation can address several pain points in the sales process:

1. Eliminating manual copying and pasting of information into a new document:

2. Creating brand consistency: Irregular documents with inconsistent branding become a thing of the past with the help of document generation. Instead, template versions can be managed and often predefined, ensuring the correct and current version is distributed to customers.

3. Gaining more visibility into documents: Many document-generation solutions simultaneously save a copy of the document directly in the organization’s CRM, enabling everyone on the team to see the date and time the document was created and sent, as well as reference the document contents, allowing for more visibility across departments and organizations.

4. Optimizing the sales process: Document-generation technology is a key piece in creating a streamlined sales workflow, or one requiring minimal manual and administrative interventions from sales reps.

Shorter Sales Cycles

Integrated solutions typically offer an easy way to configure and provide a seamless, highly professional front-end experience for both reps and customers.

More importantly, companies that effectively streamline sales processes increase their company revenue at more than two times the rate of all others, year-over-year. These companies also shorten the length of their sales cycle, while others experience longer sales cycles, according to the Aberdeen Group.

Document generation plays a central role in the sales cycle and has to be taken into account when planning how sales enablement can empower sales teams and lead to sales process optimization and efficiency.

Through the support of document generation, automating manual processes will give time back to the sales team and allow them to focus on the most important aspect of selling: directly engaging with customers and prospects.

Will Spendlove is vice president of product marketing at
Conga. A veteran marketer, with experience spanning more than 18 years in the technology space, he specializes in product marketing and sales enablement. He served as head of product marketing at SteelBrick and InsideView, and for the Salesforce Remedyforce product. He also has run sales enablement teams at Symantec, OpenText and Embarcadero technologies.

Let’s block ads! (Why?)

CRM Buyer

Think Agile Can’t Work With ERP? 7 Common Myths Debunked

July 20, 2019 BI News and Info

Part 1 in the 3-part “Agile ERP” series

Agile development is a proven methodology that has been adopted by leading software firms all around the world that have been convinced of the benefits it can deliver. So why is it that I come across so many people who, despite the evidence to the contrary, seem to believe modern application delivery processes like agile and DevOps simply don’t, or can’t, apply to arguably the most critical applications in their business – their enterprise resource planning (ERP) suite?

It’s a topic that’s on my mind a lot at the moment, as I’ve been working with several companies that have seen tremendous benefits following a transition to agile development in their ERP applications.

The fact is that many modern IT organizations are under enormous pressure to deliver business benefits more quickly, and old ways of working simply no longer cut it. I’ve already talked about the benefits and value an agile model can bring to ERP software, so I’m going to take the opposite view here and explain why some of the most common myths and misconceptions regarding agile ERP don’t hold water.

Myth #1: Stability and resilience are key. Agile just adds risk.

ERP systems have been built on stability and resilience, and as such, there’s a common perception that they shouldn’t be changed quickly. This perception is fundamentally driven by the fear (and consequences) of breaking things.

Of course, it’s true that ERP is traditionally regarded as a “system of record,” but in many cases, it’s also the engine that directly enables more visible “systems of engagement,” such as websites and mobile apps. Businesses and their customers are demanding more and more from such systems of engagement, which are constantly changing in response. ERP systems need to move at a corresponding pace if they can continue providing the valuable differentiation that sets a business apart from its peers.

Agile processes and tools enable ERP developers to meet the apparently contradictory requirements for system resilience and rapid change. The smaller, incremental changes seen in agile are actually far less risky to manage and deploy.

Myth #2: Agile can’t provide the compliance, documentation, and approvals we need.

OK, there’s no doubt that certain industry sectors like healthcare or government, require a lot more compliance, governance, and documentation. At first glance, this may seem to be at odds with an agile approach, given that the Agile Manifesto explicitly gives more weight to software than documentation. In reality, that’s not the case.

Agile makes engineering and development processes much more efficient, which means that the right solutions can be delivered more quickly – a beneficial outcome even in regulated industries where specific documentation and compliance are required.

Regardless of mandated processes, you’re more likely to be testing, approving, and documenting things sooner with an agile approach, especially if you embed the right amount of regulation and industry knowledge in the development team. With appropriate processes and automated tooling, compliance might even become simpler and less time-consuming.

This accelerated pace of change also allows organizations to respond more quickly when legislation changes. That’s potentially a commercial advantage and certainly a way to reduce risk.

Myth #3: Waterfall works. There’s no need to reinvent the wheel.

The waterfall methodology – where requirements are fully defined upfront and implemented in one big large chunk – has been the bedrock of many an ERP implementation project, and in many cases, has worked perfectly well. So why change something that everybody is familiar with and is proven to be effective (at least in some cases)?

Well, not to put too fine a point on it, waterfall often doesn’t work. Or at least not as well as it should. Projects are frequently slow and over budget. The long delivery timeline encourages scope creep and at the same time reduces the chance that the final product will meet customer needs. Even the pressure of a single hard cutover deadline can be counterproductive, encouraging deployment of unfinished code to stay on schedule.

Agile can address all of these issues by delivering frequent iterations, clear prioritization, and constant feedback. I’ve seen agile approaches work very well in some huge organizations that run ERP, although it can admittedly take time to get it right.

Myth #4: A hybrid waterfall + agile approach is better.

There is often a reluctance to take steps towards agile without definitive proof that it works: a chicken-and-egg situation. Comfort and familiarity with the kind of fixed-price, pre-budgeted ERP projects I’ve just mentioned can also be a barrier.

One solution in this situation may be to “sample” agile in specific parts of the development process via a “water-agile-fall” approach. Here, the build phase of the project is carried out in iterations while the design phase and full integration testing proceed in the traditional manner. This hybrid approach can be used to retain the familiar upfront planning and budgeting and the downstream testing and release afterward.

It’s a seductive idea: all of the best bits of the two approaches without the downside.

While there will always be outlying examples where this kind of hybrid approach remains most appropriate, in reality it means your business will never appreciate the full value of agile. It should be a temporary solution that provides the evidence to support a full roll-out.

A key feature of ERP is a large amount of integration between business processes, which may be distributed across different teams and functional areas with limited collaboration. This complex situation often reinforces the idea that agile can’t work.

However, a major success factor in such a situation is a clear understanding of the dependencies across the solution, so that they can be managed effectively, and the focus remains on the required business outcomes.

Part of this understanding comes from a clear view of how teams are structured, so it’s vital to have business and architectural knowledge as part of the solution delivery.

Agile really helps in this situation. Effective backlog management and sprint planning ensure that dependencies between processes and teams can be well-managed. This results in requirements that are implemented in the right order and ensures that solution integrity is maintained.

Myth #6: Agile doesn’t work for greenfield implementations.

It is often said that agile can work in ERP, but not if yours is a greenfield implementation or you’re rolling out a major program of work where there’s just too much to manage.

It’s true that in this case, the project could last many months or even years before a solution is delivered for use in production. But that doesn’t mean that the business should have to wait months before they get their eyes in front of and hands onto what’s being built. That’s a recipe for failure.

In this scenario, agile business managers are integrated into the process from the start so they can act as product owners and constantly steer development and configuration. They also get to see and test what’s been built at an early stage. This avoids the lengthy development of sub-optimal solutions that might be seen in waterfall approaches where the business has no visibility of outcomes until final project delivery.

Myth #7: Agile doesn’t work with outsourced and offshore teams.

In many ERP organizations, teams are distributed and often comprise a mix of outsourced vendors. This can prove a challenge to the adoption of agile due to the perception that communication will be hampered if team members are not in the same room.

The first step in jumping over this hurdle is simply for teams and partners to be open to agile methods. This might already be the case, or it might need some work, but this attitude is becoming commonplace for most vendors since support for agile and similar methods is necessary to remain competitive.

In fact, agile can actually be a benefit in this distributed model. Daily scrums, along with planning and retrospective meetings, ensure that all team members are aligned with the product owner. They understand priorities and dependencies and have constant communication and checkpoints, enabling risks to be managed more effectively. This actually alleviates a key risk of the offshore model: communication breakdown.

Conclusions

Resistance to change is hardly an uncommon phenomenon. It’s understandable that the move to agile will be uncomfortable for some, especially in ERP where it still hasn’t been seen very often.

To make agile work, you need to have the right people with a mix of functional and technical skills, who are open to working in a different way, and can be trusted to do just that. It will take time to adapt, but it can be done!

Although ERP projects are often long and complex, in many cases, the more there is to deliver, the better an agile approach can be. Agile excels in complex systems like ERP, where there are many things that can’t be anticipated, as a method for dealing with the stuff you simply can’t plan.

If you’re interested in agile for ERP – or maybe this post has convinced you that it’s worth looking into – take a look at this e-book for some tips to help you get started.

This article originally appeared on the Basis Technologies blog. This adapted version is republished by permission. Basis Technologies is an SAP silver partner.

Let’s block ads! (Why?)

Digitalist Magazine

Are You a Digital Disruptor?

July 20, 2019 TIBCO Spotfire

Digital disruptors are built around digital technology and innovation. They operate and adapt faster than their competitors. They automate decision making so that they don’t miss opportunities that appear and disappear within seconds. They create game-changing user experiences that put all of their businesses’ services in the hands of their customers. And they leverage data as a business asset to inspire innovation.

Technology allows these disruptive businesses to emerge quickly in any industry, and rapidly attract customers from current market leaders with new superior offerings. Consider companies like Uber, Stripe, Twilio, and AirBnB—each took advantage of digital services and the cloud to create innovative new business models that disrupted incumbent businesses that are decades old.

Will your business be a digital disruptor? Or will it get disrupted? There are a number of roadblocks that can hold you back from becoming a disruptor in your industry.

Your Current Information Technology Architecture is Expensive and Complex

Some information technology architectures are decades old and composed of expensive platforms and systems that require specialized skills to maintain. These architectures make it hard for users to consume IT services because they are difficult to understand and utilize. What if there was a way to create a simple view of your architecture that makes your IT services easier to consume across your business ecosystem? If you want to be a digital disruptor—outdated technology won’t get you there.

You Don’t Have Sufficient IT Staffing

Digital disruption is about more than just technology—it’s about empowering people with tools to help them collaborate and innovate. They need to quickly connect information assets together to uncover meaningful insights. Unfortunately, if these connections can only be made by your IT staff, then your business cannot operate at the speed of a digital business, as the backlog of IT requests usually far outstrips the staff available to service them. But what if your IT services were standardized so that easy-to-use tools could help non-technical users quickly connect them together themselves? To become a digital disruptor, you have to empower everyone to connect anything so that everyone can innovate.

You Risk Breaking Your Business When You Introduce Change

Change is always risky. When you evolve your IT environment to take advantage of new architectures and technologies, you risk negatively impacting the people and processes that depend on it. But what if you could evolve your IT environment so that your architecture and services embraced change without breaking your business? Digital disruptors not only operate with speed, they also adapt quickly to change.

Are you ready to become a digital disruptor? Read our new eBook to find out how you can overcome these common roadblocks and take a leap forward in your industry.

Let’s block ads! (Why?)

The TIBCO Blog

AIN’T THIS THE TRUTH?

July 20, 2019 Humor

Let’s block ads! (Why?)

ANTZ-IN-PANTZ ……

Remover Horário de Verão no Brasil em 2019 [Remove Daylight Saving Time in Brazil in 2019]

July 20, 2019 Microsoft Dynamics CRM

Olá, pessoal,

Através do Decreto Presidencial número 9772 de 25/04/2019 o Horário de Verão no Brasil em 2019 foi encerrado (http://www.planalto.gov.br/ccivil_03/_ato2019-2022/2019/decreto/D9772.htm). Desta forma, se faz necessário atualizar as configurações dos bancos de dados das organizações Dynamics CRM/365 para ignorar o Horário de Verão previamente estabelecido. Elaborei este script para ser executado nos ambientes On Premises a partir da versão 2011. O script gerará uma cópia da tabela TimezoneRuleBase, caso ha alguma mudança no Horário de Verão futuramente. Na primeira linha ajustar o nome do banco de dados da Organização a atualizar.

IMPORTANTE: (1) Este script trata somente da atualização do banco de dados do Dynamics (server) e não inibe a necessidade de updates do Windows (server / client).

IMPORTANTE: (2) Para os ambientes Online a Microsoft aplicará o ajuste em todas as organizações automaticamente.

USE [nome_da_organizacao_MSCRM] – alterar para o banco de dados da organização
GO

BEGIN TRY

BEGIN TRANSACTION

IF (NOT EXISTS (SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = ‘TimezoneRuleBase_Backup’))
BEGIN
SELECT * INTO TimezoneRuleBase_Backup FROM TimezoneRuleBase
END

UPDATE TimeZoneRuleBase
SET
StandardBias = 0,
StandardYear = 0,
StandardMonth = 0,
StandardDay = 0,
StandardDayOfWeek = 0,
StandardHour = 0,
StandardMinute = 0,
StandardSecond = 0,
DaylightBias = -60,
DaylightYear = 0,
DaylightMonth = 0,
DaylightDay = 0,
DaylightDayOfWeek = 0,
DaylightHour = 0,
DaylightMinute = 0,
DaylightSecond = 0
WHERE TimeZoneDefinitionId in (SELECT TimeZoneDefinitionId FROM TimeZoneDefinitionBase WHERE TimeZoneCode IN (65, 58))
AND EffectiveDateTime = ‘2019-01-01′

COMMIT TRANSACTION

END TRY

BEGIN CATCH

ROLLBACK TRANSACTION
THROW

END CATCH

Walter Carlin – Sr. PFE – Dynamics 365 – Microsoft

Let’s block ads! (Why?)

Dynamics 365 Customer Engagement in the Field

AI Weekly: A growing chorus of experts agrees facial recognition systems must be regulated

July 19, 2019 Big Data

On Tuesday, Oakland became the third U.S. city after San Francisco and the Boston suburb of Somerville to ban facial recognition use by local government departments, including its police force. The ordinance adopted by the city council, which was written by Oakland’s Privacy Advisory Commission and sponsored by Councilmember Rebecca Kaplan, prohibits the city and its staff from obtaining, retaining, requesting, accessing, or using facial recognition technology or any information gleaned from it.

Oakland’s ban was long expected — the Oakland Privacy Advisory Committee endorsed the legislation’s wording late last month — but it comes as a growing chorus of AI experts, privacy advocates, and lawmakers express concerns over the largely unregulated tech’s applications.

A September 2018 report revealed that IBM worked with the New York City Police Department to develop a system that allowed officials to search for people by skin color, hair color, gender, age, and various facial features. Elsewhere, the FBI and U.S. Immigration and Customs Enforcement are reportedly using facial recognition software to sift through millions of driver’s license photos, often without a court order or search warrant. And this past summer, Amazon seeded Rekognition, a cloud-based image analysis technology, to law enforcement in Orlando, Florida and the Washington County, Oregon Sheriff’s Office. The City of Orlando said this week it discontinued its Rekognition pilot, citing a lack of necessary equipment or bandwidth. But Washington County used Rekognition to build an app that lets deputies run scanned photos of suspected criminals through a database of 300,000 faces, which the Washington Post claims has “supercharged” police efforts in the state.

Nonprofit advocacy group Fight for the Future yesterday published a map highlighting the speed with which facial recognition is spreading. The map lists dozens of U.S. airports, state and local law enforcement agencies, states, and cities where such systems are in active use.

Discouragingly, as experts have repeatedly noted, there appears to be little correlation between facial recognition systems’ respective accuracy and the pace of their deployment. It was recently revealed that a system used by London’s Metropolitan Police produces as many as 49 false matches for every hit. During a House oversight committee hearing on facial recognition technologies in 2017, the U.S. Federal Bureau of Investigation admitted that the algorithms it uses to identify criminal suspects are wrong about 15% of the time. And MIT Media Lab researcher and Algorithmic Justice League founder Joy Buolamwini discovered in audits of facial recognition systems — including those made by Amazon, IBM, Face++, and Microsoft — that they performed poorly on young people, women, and people with dark skin.

The evidence has led analysts like Clare Garvie, a senior associate at the Georgetown University Center on Privacy and Technology and coauthor of the Perpetual Lineup report, which monitors trends in computer-assisted facial recognition, to conclude that facial recognition technology could cause extraordinary harm. Last month, the center released reports detailing the NYPD’s use of altered images and pictures of celebrities who look like suspects to make arrests, as well as real-time systems being used in Detroit and Chicago and tested in other major U.S. cities.

“Imagine if we had a fingerprint lab drawing fingerprints or drawing a latent print’s finger ridges with a pen and submitting that to search,” she told the House Oversight and Reform Committee in May. “That would [be] a scandal, that would be a reason for a mistrial or convictions being overturned, and it’s hugely problematic.”

Perhaps unsurprisingly, beyond the adoption of outright bans, lawmakers at the national, state, and local levels have pushed back against unfettered facial recognition software. U.S. Congress House Oversight and Reform Committee hearings in May saw bipartisan support for limitations on systems use by law enforcement, and state legislatures in Massachusetts and Washington have considered imposing moratoriums on face surveillance platforms. Separately, the California State Legislature is currently weighing a facial recognition ban on police body cam footage, as is the Berkeley City Council.

But calls for regulation and cooperation with watchdog groups haven’t exactly been universal. When Georgetown researchers first requested facial recognition records from the NYPD in 2016, they were told that no such records existed — despite the fact that the technology had been in use since 2011. Only after two years in court did the agency turn over 3,700 pages of documents related to its facial recognition software use.

Garvie and others say that in future legislation, they’d like to see mandatory bias and accuracy testing, court oversight, minimum photo quality standards, and public audits (like the annual surveillance tech use reports already required in San Francisco). Critics also advocate against real-time facial recognition use by police and the practice of scanning driver’s license databases with facial recognition software. And many believe that prosecutors and police should be obligated to tell suspects and their counsel if facial recognition aided in an arrest.

It’s a lengthy checklist, but Garvie believes it’s the baseline required to deploy facial recognition technology responsibly.

“What we’re seeing today is that in the absence of regulation, [facial recognition] continues to be used, and now we have more information about just how risky it is, and just how advanced existing deployments are,” she said in a previous statement. “In light of this information, we think that there needs to be a moratorium until communities have a chance to weigh in on how they want to be policed and until there are very, very strict rules in place that guide how this technology is used.”

Thanks for reading,

Kyle Wiggers

AI Staff Writer

P.S. Please enjoy this segment from CBS about how facial recognition technology is outpacing the law.

https://www.youtube.com/watch?v=__De_2N3nWM

Let’s block ads! (Why?)

Big Data – VentureBeat

Modeling Time

July 19, 2019 BI News and Info

Time is an interesting thing. It moves in only one direction; it’s units of measurement are irregular and constantly in motion. When we go to model it in SQL, it’s usually easier to do lookup tables (calendars) rather than trying to compute it. SQL is not a computational language, but it also must deal with the nature of this dimension. Digital computers are based on discrete states, but time by its very nature is a continuum. In a continuum, there is always an infinite number of points between any two points in the continuum. This means we’re already in a state of heresy when we try to put time into a digital computer.

Quantum Models

The quantum model is the simplest model, and a timestamp is the most common example. We simply ignore the inconvenience of the continuum and assume that time comes in discrete units or quanta. These quanta are sometimes called “chronons” in the literature. If you want to have a physical instrument for this model, think of a wall clock whose hands jump from one number to another, without sweeping in between them on the face of the clock. Or think about a calendar, whose unit of granularity is a day.

If you are a fan of kids’ television shows, you might have seen an iCarly episode that featured a new integer between 5 and 6, called “derf.” The same gimmick was recently used in an episode of DUST, a science fiction series on YouTube, if you do not to want to admit you watch kids’ television shows. Whether you treat the idea as joke or a scary fantasy, the point is that a temporal model has to be based on discrete values without gaps and a strong ordering for computations.

But what happens when you have to record a temporal value from the real world which is between quanta with such a scale? You have to assign or ignore a temporal value that is in the set of allowed values. In the Orient, a person’s age in years is a count of the full and partial years that they have lived. A new-born baby is one year old in this system. In the Western world, we only count the full years and use “months” if the child has lived less than a year. Or, to put it another way, you can round a temporal value toward the past or toward the future to map it to a quanta value in this model.

The old DATETIME data type in Transact-SQL defines a date that is combined with a time of day with fractional seconds that is based on a 24-hour clock. The date range is from 1753 January 1 to 9999 December 31. The start date is the beginning of the Gregorian calendar when it was first adopted in Great Britain. The end date is the largest four-digit year that can be represented on it. Other countries adopted it latter times, often years later (for a very detailed history, read THE CALENDAR by David Ewi ng Duncan, ISBN 9781857029796). Today we’re not really on the Gregorian calendar anymore. We use what’s called the Common Era calendar and we’ve replaced A.D. and B.C. with CE and BCE. It’s almost like the Gregorian calendar, except it starts at 0001-01-01 and goes through 9999-12-31.

The time range is 00:00:00 through 23:59:59.997, but the fractional seconds are weird. They are rounded to quanta of 0.000, 0.003, or 0.007 seconds. The reason for this has to do with the original implementation of SQL Server under Sybase. DATETIME data was stored in the floating-point format available on the original 16-bit hardware. The mantissa held the date portion, and the exponent held the clock portion. This is why if you look at old SQL Server code you may find a DATETIME value CAST or CONVERTed to floating-point, manipulated and then cast back to DATETIME. If this sounds like a kludge, that’s because it is. But it was a very fast kludge, especially if your machine had floating-point hardware. Since we had no separate DATE and TIME data types, the convention was to fake having a DATE by setting the time portion to 00:00:00.000. You will find a lot of old code for this in legacy systems.

SQL Server has had ANSI/ISO compliant date and time datatypes for several years now. You should not ever use the old DATETIME data type, and instead you should be moving on to DATETIME2(n) and DATE datatypes today. I also think it’s probably a good idea to go through and replace those old kludges with the modern datatypes. The new datatypes are not only ANSI/ISO 8601 conformant but use less storage and have more precision if you really need it. Let’s be honest, most of the time, commercial work is quite happy with precisions to a date or to a fraction of an hour within the date. While it’s nice to be able to go down to nanoseconds, we really don’t use that degree of precision in any common application you will ever run into.

Another advantage is using the ISO 8601 format which is an international standard with unambiguous specification. Also, this format isn’t affected by the SET DATEFORMAT or SET LANGUAGE setting. This format has several display options; SQL picked one.

Today, the ANSI/ISO SQL Standards only allow the ISO 8601 format which looks like “yyyy-mm-dd HH:mm:ss.ssssss”, in the language. Please notice the use of dashes and a space between the date and the time fields. The ISO 8601 standard also specifies a version that drops out the dashes and replaces the space with a “T” as a separator, which SQL Server also allows. Frankly I like the second version better because of the continuous string with no worries about white spaces (Blank? Tab? Newline? Carriage return?) and I’m very glad to see that Microsoft supports it. However, we picked the first version when we were setting up the standards because it looks nicer for people. It is the default in the SQL world.

Other Date Displays

Probably the most esoteric way of representing a date is the Julian system. It’s used by astronomers and it has nothing to do with the Julian calendar. Nobody ever uses that outside of the sciences. The count starts with 4713, January 01 BCE, and the Julian Date for 00:30:00.0 UT January 1, 2013, is 2 456 293.520 833. For details see this article and this one.

Another common way of displaying a date is simply the ordinal date format. It consist of “yyyy-ddd” where the “yyyy” is the usual year and “ddd” is the day within the year, taking on the values 001 thru 365 or 366.

Closed Interval Models

Another model is based on a closed interval defined by (start_timestamp, end_timestamp) pairs that mark the end points of a temporal interval. We assume (start_timestamp ≤ end_timestamp). All of the temporal points in the interval are assumed to be in the set of values, even if you cannot actually represent them with your software. The FIPS (Federal Information Processing Standards) require at least four decimal places of seconds in a timestamp in all clocks used by the US government. However, the current hardware and SQL Standards can give seven decimal places, nanoseconds.

A calculus for temporal reasoning was introduced in 1983 by James F. Allen (“Maintaining knowledge about temporal intervals”, Communications of the ACM. 26(11): 832–843.). It was not designed for SQL, but for general use when describing events. The following 13 base relations capture the possible relations between two intervals in Allen’s Interval Algebra:

word image 48 Modeling Time

The inverse of a relation simply flips the right and left sides of the infixed operators and rewords the definition. Obviously, equality is its own inverse.

In general, the number of different relations between n intervals is 1, 1, 13, 409, 23917, 2244361… OEIS A055203. The special case shown above is for n=2.You have to decide if endpoints are co-located, or merely tangent. It gets messy, but it is a consistent, known model.

ISO Half Open Interval Model

The ISO model of time is based on half open intervals. That means we have a starting point, but we have no ending point for an interval. The duration in the interval approaches but never actually reaches the ending point. To make this a little more explicit, consider a day. It begins at 00:00:00 Hrs (midnight), progresses through 24 hours, but never gets to 24:00:00 Hrs. That final endpoint actually belongs to the start of the next day. DB2, other SQL products, and international conventions will convert 24:00:00 Hrs to midnight of the next day automatically.

Depending exactly in what precision your times are kept, you can get as close as 23:59:59.9999999 Hrs, assuming that you’re following the precision required by Federal Information Processing Standards. In actual practice, for commercial purposes, you probably are not paying your employees by the nanosecond. Let’s be honest, to be within one minute is probably good enough in most cases.

The advantage of the half open interval model is that two time periods can be abutted to each other. Each point in time belongs to one and only one interval. If you used closed intervals, then they would abut on at least one point, or could overlap. This makes for a lot of problems when you are trying to determine during which shift a particular billable event occurs.

In mathematics, this is called the partitioning of a set, and it’s where we get the name of the PARTITION BY clause in SQL. But unlike mathematics, SQL (or any other digital representation in a computer) can’t actually be a continuum. To give you an idea where I’m going, let’s assume that a day is broken into three shifts in a factory for purposes of timecards.

CREATE TABLE Shifts

(

shift_code CHAR(12) NOT NULL PRIMARY KEY

CHECK (shift_code IN (

‘first_shift’, ‘second_shift’, ‘third_shift’ )),

start_time TIME(0) NOT NULL,

end_time TIME(2) NOT NULL,

CHECK (start_time < end_time)

);

INSERT INTO Shifts

VALUES

(‘first_shift’, ’09:00:00′, ’16:59:999′), —nine to five

(‘second_shift’, ’17:00:00′, ’19:59:999′), —five to eight

(‘third_shift’, ’20:00:00′, ’23:59:59.999′); – graveyard shift

CREATE TABLE Timecards

(

emp_name VARCHAR(25) NOT NULL PRIMARY KEY,

clock_in_time TIME(0) NOT NULL,

clock_out_time TIME(0) NOT NULL,

CHECK (clock_in_time <= clock_out_time)

...

);

This is a pretty skeletal timecard system created for illustrating the basic principles of this article. A real schema would have a lot more details.

For example, employee A enters a time in/out block 08:30:00– 17:30:00. That block would have to be parsed as something like

SHIFT CODE third_shift = 0.5 hours

SHIFT CODE first_shift = 8 hours

SHIFT CODE second_shift = 0.5 hours

We can use decimal representation for hours… common in payroll and time tracking. Many decades ago, I had a similar problem. We rounded the durations to 15 minute blocks (xx:00:00 thru xx:14:59, xx:15:00 thru xx:29:59, xx:30:00 thru xx:44:59, xx:45:00 thru xx:59:59) as per union rules. 24 hrs per day * 4 blocks per hour = 96 blocks per day, 5 days per work week = 480, 50 work weeks per year = 24,000; a decade of lookups = 240,000 rows. Put a running total of blocks in each row, then subtract the starting block count from the ending block count.

We found that a simple lookup table was easier than trying to do temporal math. After all, SQL is a data language, not a computational language. Use a spreadsheet and load the table, so you are done for a decade.

The OVERLAPS() Predicate

ANSI/ISO Standard SQL defines the OVERLAPS() predicate as the result of the following expression:

(S1 > S2 AND NOT (S1 >= T2 AND T1 >= T2))

OR (S2 > S1 AND NOT (S2 >= T1 AND T2 >= T1))

OR (S1 = S2 AND (T1 <> T2 OR T1 = T2))

where S1 and S2 are the starting times of the two time periods and T1 and T2 are their termination times. The rules for the OVERLAPS() predicate sound like they should be intuitive, but they are not. The principles that we wanted in ANSI/ISO Standard SQL were:

1. A time period includes its starting point but does not include its end point. We have already discussed this temporal model.

2. If the time periods are not “instantaneous”, they overlap when they share a common time period.

3. If the first term of the predicate is an INTERVAL and the second term is an instantaneous event (a <datetime> data type), they overlap when the second term is in the time period (but is not the end point of the time period). That follows the half-open model.

4. If the first and second terms are both instantaneous events, they overlap only when they are equal.

5. If the starting time is NULL and the finishing time is a <datetime> value, the finishing time becomes the starting time and we have an event. If the starting time is NULL and the finishing time is an INTERVAL value, then both the finishing and starting times are NULL.

Please consider how your intuition reacts to these results, when the granularity is at the YEAR-MONTH-DAY level. Remember that a day begins at 00:00:00 Hrs.

(today, today) OVERLAPS (today, today) = TRUE

(today, tomorrow) OVERLAPS (today, today) = TRUE

(today, tomorrow) OVERLAPS (tomorrow, tomorrow) = FALSE

(yesterday, today) OVERLAPS (today, tomorrow) = FALSE

Keeping Contiguous Events

Alexander Kuznetsov wrote this programming idiom for History Tables in T-SQL, but it generalizes to any SQL. It builds a temporal chain from the current row to the previous row with a self-reference. This idiom easy to use, once the DDL is in place.

This is easier to show with code:

CREATE TABLE Tasks

(

task_id INTEGER NOT NULL,

task_score CHAR(1) NOT NULL,

previous_end_date DATE, – null means first task

current_start_date DATE

DEFAULT CURRENT_TIMESTAMP NOT NULL,

CONSTRAINT previous_end_date_and_current_start_in_sequence

CHECK (previous_end_date <= current_start_date),

–DEFERRABLE INITIALLY IMMEDIATE,

current_end_date DATE, – null means unfinished current task

CONSTRAINT current_start_and_end_dates_in_sequence

CHECK (current_start_date <= current_end_date),

CONSTRAINT end_dates_in_sequence

CHECK (previous_end_date <> current_end_date),

PRIMARY KEY (

task_id,

current_start_date

UNIQUE (

task_id,

previous_end_date

), – null first task

UNIQUE (

task_id,

current_end_date

), – one null current task

FOREIGN KEY (

task_id,

previous_end_date

) – self-reference

REFERENCES Tasks (

task_id,

current_end_date

)

);

Well, that looks complicated! Let’s look at the code column by column. Task_id explains itself. The previous_end_date will not have a value for the first task in the chain, so it is NULL-able. The current_start_date and current_end_date are the same data elements, temporal sequence and PRIMARY KEY constraints we had in the simple history table schema.

The two UNIQUE constraints will allow one NULL in their pairs of columns and prevent duplicates. Remember that UNIQUE is NULL-able, not like PRIMARY KEY, which implies UNIQUE NOT NULL.

Finally, the FOREIGN KEY is the real trick. Obviously, the previous task must end when the current task started for them to abut, so there is another constraint. This constraint is a self-reference that makes sure this is true. Modifying data in this type of table is easy but requires some thought. The foreign key reference is to be disabled when this table is first constructed and then restarted afterwards. In ANSI/ISO Standard SQL, this is done with deferrable constraints, but in T-SQL you have to explicitly turn the constraint on and off during the session.

DATETIMEOFFSET

DATETIMEOFFSET is a data type giving you the ability to work with time zones. Here’s the format:

YYYY-MM-DD hh:mm:ss[.nnnnnnn] [{+|-}hh:mm]

The final option on this date format is the number of hours and minutes that the time zone difference differs from UTC. Time zones can be a little crazy, usually for political reasons rather than computational considerations. For example, for many years Japan and Korea had different zones. Columbia did not want to be in the same time zone as United States when Caesar Chavez was their dictator. You can find the currently recognized time zones here. SQL Server doesn’t care if the time zone offset exists in the real world or not, it just needs to be computationally valid.

To generate the current local non-offset date and time value in the target SQL Server instance, you use functions such as SYSDATETIME (returns DATETIME2) or GETDATE (returns DATETIME). To generate the current local date and time value with the offset, use SYSDATETIMEOFFSET. This example returns the date and time, including the offset, for US Central Standard Time when running on my computer:

PRINT SYSDATETIMEOFFSET();

This command returns:

word image 49 Modeling Time

To return the current time zone offset, you use functions like DATENAME (returns a string with hours and minutes offset) or DATEPART (returns an integer with minutes offset) with the TZoffset part (tz in short). The displacement will make allowances for daylight saving time, according to Microsoft conventions. Here’s an example which returns -300 since the offset is -5 hours:

DECLARE @Date DATETIMEOFFSET = SYSDATETIMEOFFSET();

PRINT DATEPART(tz,@Date);

AT TIME ZONE

The AT TIME ZONE function was introduced in SQL Server 2016. It replaces both TODATETIMEOFFSET and SWITCHOFFSET. It has the following syntax:

<value> AT TIME ZONE ‘<standard time zone name>’

Here’s an example:

DECLARE @Date DATETIMEOFFSET = GETDATE();

PRINT @Date;

SET @Date = @Date AT TIME ZONE ‘Pacific Standard Time’;

PRINT @Date;

word image 50 Modeling Time

When the input value is a nonoffset date and time value, the function behaves similar to TODATETIMEOFFSET; when the input value is a date and time value with an offset, the function behaves similar to SWITCHOFFSET. Moreover, you don’t need to worry about clock switching; rather, just specify the target standard time zone name (for instance, for Pacific Time always specify ‘Pacific Standard Time’), and SQL Server will figure out dynamically the target time zone offset based on the Windows time zone conversion rules. To get the full list of supported standard time zone names, plus their current offset from UTC and whether it’s currently on daylight saving time, query the sys.time_zone_info function:

SELECT * FROM sys.time_zone_info;

The query returns 139 rows, but here is a sample:

word image 51 Modeling Time

I also strongly recommend that you read this article, so have some idea just how complicated things can get.

The Most Important Thing to take Home from this article

Use the TIME, DATE, DATETIME2 and DATETIMEOFFSET data types for new work. These types align with the SQL Standard. They are more portable. DATE, TIME DATETIME2 and DATETIMEOFFSET provide more decimal seconds precision. DATETIMEOFFSET provides time zone support for globally deployed applications, but think in UTC.

Let’s block ads! (Why?)

SQL – Simple Talk

FY20 Microsoft Business Applications Inner Circle Members List Includes 7 CRM Software Blog Members

July 19, 2019 CRM News and Info

Congratulations to the FY20 Microsoft Business Applications Inner Circle Members, including seven CRM Software Blog members.

According to Microsoft the Business Applications Inner Circle partners represent the top 1% of the total Business Applications ecosystem and drive more than 30% of the FY19 Worldwide Business Applications Cloud Revenue.

Inner Circle Winners – CRM Software Blog Members:

More Inner Circle Winners:

adesso AG
AlfaPeople
Alithya
Alterna
Annata
Arbela Technologies (ERP Software Blog member)
Armanino LLP
Avanade
Avtex Solutions, LLC
Axxon Consulting
Barhead Solutions
Blue Horseshoe
Capgemini
Capita
CEC
Cegeka
Columbus A/S (ERP Software Blog member)
Controles Empresariales
COSMO CONSULT Group
Delegate
Cy
DXC Technology
eCraft Oy Ab
eLogic
EOS Solutions Group
Experlogix, Inc.
EY
Fellowmind
Fusion5
Hitachi Solutions
HSO
Incremental Group
Intergen
ITVT Group
KPMG
LS Retail
MASAO
Mazik Global
MCA Connect
Mint Group
NTT Data & Sierra Systems
ORBIS AG
PowerObjects an HCL Technologies Company
PrenticeWorx
Prodware
PROS
PwC
QBS Group
QUANTIQ
Right Group
SAGlobal
Sikich
Sunrise Technologies (ERP Software Blog member)
Tectura Hong Kong Limited
Thinkmax Consulting Inc.
To-Increase (ERP Software Blog member)
Velrada
VeriPark
XPLUS

For the first time, to select the FY20 members of The Business Applications Inner Circle, Microsoft took into account only Cloud Billed Revenue (with the exception of top Business Central partners) and Dynamics 365 Customer Adds. Microsoft also took into consideration these partners’ adoption of the broader Microsoft Cloud portfolio (IoT, AI, etc.), the importance of these partners in their local market and their investments in differentiated solutions offerings.

We congratulate all of these companies on their accomplishment.

By CRM Software Blog Writer, www.crmsoftwareblog.com

Let’s block ads! (Why?)

CRM Software Blog | Dynamics 365

What’s Your Spend Management Costing You?

July 19, 2019 BI News and Info

You’ve hit your spending limit. Using outdated manual, paper-based systems to track expenses is costing your organization money and your own employees’ sanity. With employees spending more money across more channels than ever before, the difficulty of keeping track of spend as an organization is skyrocketing. Answer the following question to see where your own organization currently stands:

Of the list below, what do you want to be true for your company?

Streamlined systems
Maximized productivity
Accurate spend data
Enabled and satisfied employees
Efficient spending
All of the above

We’re going to take a wild guess and say you picked all of the above. The way you decide to set up your expense process affects every facet of your business. And that means manual expense reports are taking a bigger toll on your company than you might think.

The true cost of spreadsheets

Doing things the hard way is also the expensive way. It costs 39% of businesses more than US$ 6 to process a single invoice. Multiply that by the number of expense reports you handle each month, and you could be looking at a pretty substantial hit. And that’s just the tip of the iceberg:

What does the average expense report cost for your organization? Let’s do the math: The number of hours spent on expense reports multiplied by the average hourly salary of your staff, divided by the number of expense reports you process per year, will give you your organization’s customized number. It looks like this:

(Hours spent annually on expense reports) x (Average hourly rate of your processing staff) ÷ (Number of expense reports per year) = Cost per expense report

1561042747965 costperreport whitebg What’s Your Spend Management Costing You?

Now that you know how much your expense report is costing you, let’s dive into how automation can help.

Stay streamlined

You can happily say goodbye to inefficient, manual processing and say hello to an automated and connected travel, expense, and invoice solution. With automation on your side, your organization can confidently operate with reliable and streamlined expense data. Removing unnecessary steps such as printing out documents and manually filling out information will not only benefit today’s processes but multiply in value as your organization grows and changes. Still not convinced? Then ask yourself these four questions:

Is your accounts payable invoice system scalable for growth?
Could your expense process be simplified with a streamlined solution?
Are your employees and stakeholders getting frustrated with your process?
Do you have the necessary technology support needed to keep your invoice system running seamlessly?

Maximize productivity

As employee spending becomes more complex, the manual reporting process gets even more painful, making your reports a recipe for wasted hours, days, and dollars. This translates into finance and AP personnel spending a significant amount of their time with manual tasks, while financial decision-makers spend a lot more of their time trying to analyze and interpret data. Hours spent x employee hourly wage = Cost of manual process

1561042632233 costofmanualprocesses whitebg What’s Your Spend Management Costing You?

With the help of automated, integrated solutions, you can move many manual, frustrating, and mundane tasks off your employees’ plates and free up more time for yourself and others to focus on strategic tasks that promote growth. Attaining efficiencies in AP workflows alone results in spending 68% less time processing invoices – just imagine what you could do with all that free time.

Report accurately

Everybody makes mistakes, but when 10% of invoices have errors, something needs to change. Companies that use a mixture of paper and spreadsheets to process expenses are especially prone to this problem. Paper receipts can get lost, and it’s not hard to imagine the manual errors that can crop up when copying data into a form. Manual processes are prone to inaccurate data entry and even fraudulent practices, which can result in the following:

20% of businesses say that reconciling duplicate invoices is among their top challenges with processing paper invoices, according to IOFM’s research.
Inventories can easily be off with data entry, causing overcharges—all of which can cause mistrust and confusion between you, your customers, and your employees.

Updating to automation can reduce these errors by more than 90%. By catching duplicate entries, and automatically flagging expenses outside of policy, the back-and-forth from your employee to your accounting team is virtually eliminated. All communication can be handled within the system, and completely documented for auditing purposes.

Improve employee satisfaction

Reducing your employees’ time spent manually creating and entering expenses, plus eliminating the need to keep up with paper receipts, lightens the load and boosts morale within your organization. Automating your workflows and implementing best practices for your accounting staff will significantly improve accuracy, cycle time, and employee satisfaction.

But the frustration doesn’t end with submitting expenses. Employees often wait a long time to get reimbursed, too. The average time is 8 days from approval to reimbursement, but sometimes it can take even longer. And that’s a whole lot of time spent shouldering an out-of-pocket expense. Thanks to automation, ease of use, and the availability of apps on mobile devices, various groups of employees are saving substantial time to submit and review expenses.

Easy-to-use mobile tools that provide consumer-like experiences at work connect your employees to the apps they already use at home, making booking travel, submitting expenses, and approving invoices simple and fast. When it comes down to satisfaction, it’s plain and simple: Your company will be able to effectively retain people with better systems and processes.

Maintain compliance and efficiency

Several hidden costs can fall through the cracks when using manual processes, including labor costs and fraud. Labor costs are causing many organizations to spend nearly twice as much on invoicing processes than they would with more efficient and streamlined solutions. In other words, businesses are unknowingly throwing away a lot of money.

Paper processes are also extremely vulnerable to fraud, a cost no one wants to deal with. According to ACFE’s 2018 report, the typical organization loses 5% of annual revenue to fraud each year with a total of $ 7B in reported occupational fraud loss in 2017. Sixteen percent of businesses have suffered check fraud within the past two years, according to IOFM reports. Yikes.

By utilizing AI to detect signs of fraud and built-in policy compliance, you can focus on preventing fraud rather than recovering from it. By putting a microscope on this area of spend, you can prevent fraudsters who may be active in more than one area of the business. Automation mitigates potential fraud by restricting the ability to approve invoices and initiate payments to authorized individuals. Real-time visibility into all processes prevents unauthorized individuals from accessing sensitive data. Ultimately, automated processes can help you better keep track of your organization’s money, as well as:

The great news is, you can do it!

Luckily, it’s easy to make the move from manual to automated, one-off to streamlined, and at risk to at ease.

In today’s fast-paced business climate, organizations in every sector are under pressure to do more with less. This means organizations cannot afford to squander opportunities to free up their working capital. Automating spend management with modern, easily integrated solutions just makes sense. Get clunky systems out of the way to focus on what’s driving your organization today.

This article originally appeared on Concur Newsroom.

Follow SAP Finance online: @SAPFinance (Twitter) | LinkedIn | Facebook | YouTube

Let’s block ads! (Why?)

Digitalist Magazine

How Sisense Engineered Its Cloud-Native Linux Deployment From the Ground Up

July 19, 2019 Sisense

A couple of weeks ago, we officially launched the Cloud-Native Sisense on Linux deployment after a successful beta release cycle that kick-started in Spring 2019.

As of 2017, Linux was running 90% of the public cloud workload. It is increasingly the OS of choice by enterprises and the cloud due to its many advantages: lower TCO, higher security, improved stability and reliability, flexibility, and more. Given this importance, we made it an organizational priority to invest in a Sisense on Linux deployment in late 2017.

When we sat down to plan this execution strategy, we realized there were several different ways we could approach it. For us, it was critical to not only do it right so we didn’t waste time and resources but also to deliver a product that would lead our customers into the future and support their needs in the ever-growing cloud environment.

Here’s how we re-architected Sisense with the right technologies and frameworks for the task at hand without simply porting the code over.

How Sisense Engineered Its Cloud Native Linux Deployment From the Ground Up

The First Few Months

When I was tasked with the responsibility of building a Sisense Linux deployment in late 2017, a few small steps had already been taken. Two developers had started a Linux project which initially comprised of simply porting code from one OS to the other.

They started with C/C++ code, which usually takes the longest to migrate from Windows to Linux. By the end of 2017, the team was able to show their first demo, which ran queries over the ElastiCube Manager, our high-performance database, using C/C++.

Taking Stock and Restarting the Project

Even though some more progress was made, in January of 2018, we decided to take a step back and rethink our approach to this project. Often it is less scary to take what you have and what you know and continue without questioning your approach. However, it is not always the smartest or the best course of action.

Before jumping headlong into merely porting code from one OS to the other, it was necessary to think whether it made sense to migrate all components as-is or instead, to see what language or architecture would work best for the task at hand.

We decided that where it was required and where it made sense, we would not simply port over code but rebuild the component from scratch using the most relevant stack and technology for what that component was meant to do.

There were three “buckets” in this decision-making process:

Components that would be migrated.
Components that would be rebuilt from scratch using the right framework while maintaining institutional knowledge and providing a similar user experience. For example, we concluded that several components had to be rewritten in Java. To enable this, we dedicated more than a month to training the entire engineering team in Java. We also recruited Java experts to help guide and govern the design.
Components that would stay as-is (for example, JavaScript) with very minimal changes such as updating file names and paths.

In hindsight, this was a critical decision that paved the way for a modern, enterprise-grade, full-stack analytic application that is highly-performant, reliable, and scalable. The best part is that we were able to build it in a little over a year.

The Right Technology for the Job

cloud native 2 770x433 How Sisense Engineered Its Cloud Native Linux Deployment From the Ground Up

Let’s break this down some more. The Sisense application has a few key tasks handled by different components:

1. Sisense ElastiCube or Data Engine

The Sisense ElastiCube crunches hundreds of millions of records and needs to be highly optimized. It has to be close to the OS for better control of what is being done with less overhead. Most of this code was in C and C++ and was left that way.

Takeaway: C & C++ are good to use when building highly optimized processes that are close to the OS, such as building a database engine.

2. ElastiCube Management Service and Query Service

The ElastiCube Management Service and Query Service were moved away from C# and C++ and rebuilt in Java. Java is a highly-portable and mature language that plays well in building mission-critical, high-performance applications that are CPU-intensive. The agility and complexity needed in those components are such that we needed to use a lot of frameworks that come with Java and focus only on our application logic without compromising on performance.

We already had (and continue to have) a big footprint in Node.js. It would have been easier to use Node.js everywhere. However, we resisted the urge to use Node.js everywhere and use the best language and framework for the job.

Node.js is great for responsive operations with low memory footprint. It is easier to write in Node.js and is fast to debug and develop as well. However, Java has much better performance, more caching, and long state capabilities. Java is also more suited for compilation and type checking, which is important, especially when merging releases and branches over the years. These actions can have a lot of vulnerability if not caught by compilation errors.

For example, the Management service needs to be aware of all the statuses and aware of Kubernetes with a lot of control of the systems. It made sense to build it in Java as the service needs to be efficient, highly available and multi-threaded.

On the other hand, application parts that are more tightly integrated with the UI, are easier to build it in Node.js. For example, the original pivot was implemented in C# as an IIS application. The pivot is a full stack component. It made sense to rewrite it in Node.js which allows the full stack developer to work on both the front-end and back-end in the same technology.

For web services, it’s not recommended to use C++ because the development time is too expensive. For those reasons, eventually, we decided to go with Java and, in particular, used the Spring Boot framework. We also considered a few options like Guice or EJB (which we immediately disqualified).

Takeaway: Java is useful when building mission-critical, high-performance applications that are CPU-intensive with the need for more caching, long state capabilities, and a robust set of available frameworks. Node.js, on the other hand, is useful for responsive operations with a low memory footprint and when a developer wants to work on both the front-end and back-end in the same technology (which is the genesis of Node.js).

3. Data connectors

The .NET connector-framework was replaced with a new framework based on Java because the support for .NET on Linux is via .NET Core, which was introduced in 2016, and does not contain all the functionality of the .NET framework for Windows. The connector framework acts as a pipe for transferring data. On top of this, the actual drivers for accessing most of the database providers are written in Java, so it was only natural to code the framework in Java too. The actual data crunching is done inside the ElastiCube, which is coded in C/C++.

Takeaway: Java is a natural choice for building data connectors due to its large ecosystem including database drivers and rich frameworks.

In summary, there are certain languages most appropriate for certain operations, and choosing the correct language for the operation at hand is key.

Containerizing Microservices

Another critical change in the Linux deployment was related to the architecture itself. While many components in the Windows deployment are microservice based, given the opportunity to re-architect Sisense, we decided to build a containerized microservices application using Docker for containerization and Kubernetes for orchestration.

We initially debated between Docker Swarm and Kubernetes for orchestration but decided to go with Kubernetes due to the rising popularity and the fact that Kubernetes was becoming the de-facto standard for container orchestration. While our teams were comfortable with Docker Swarm, which is considered more of the DevOps way, Kubernetes better handled other developer requirements like versioning, upgrades, releases, and rollbacks. We decided to go with Kubernetes keeping the future developer user in mind.

An interesting debate that comes with building a microservices architecture is the number of microservices you’ll break your application into.

Two years ago, we had a fairly monolith application with four or five services. That is not the case anymore. We have around 20 services today. As a rule of thumb, we try not to create too many microservices, especially ones that lengthen the call chain. It is okay to add services that are not on the call chain. In a given operation, we shouldn’t involve all the microservices in the call chain (for example, 4-5 services is okay but not all available services). It is important to remember that while microservices are a great way of building scalable and resilient applications rapidly, they also add complexity, especially with communication between them and eventually debugging. You need to find a balance between the number of microservices you create, supportability, and maintainability.

A New Way of Doing Things with Shared Storage, Updated Monitoring & Logging

Re-architecting the platform also gave us the opportunity to update old ways of doing things and create better and highly-performant new ways. For example, the Windows way of creating highly-available data is to store copies of the data on multiple servers. With this re-architecting, we were able to do away with that and rebuild that experience enabling the use of highly-distributed and available Shared Storage technologies like cloud storage providers, GlusterFS, Amazon EFS, Azure file share, Google Filestore, and many more.

Another example is logging. One of the challenges with building a microservices-based architecture is debugging because of the number of components involved and all of the different places logs can be stored. One of the first steps we took to alleviate this was to build a combined log using FluentD, which collects all the data in a centralized place. In addition, we added Grafana and Prometheus, which provide counters of what’s going on in the system by providing a detailed view of system metrics.

Learnings Along the Way

While we have come out on the other side of a successful project, the journey was not without difficulties. Some of these were challenges that we learned from and others were limitations that we have had to work with in order to provide the best experience for the end user.

1. Embracing open source technologies

We learned that embracing well-tested and mature open source technologies are game changers in how quickly and efficiently we can build a large-scale, enterprise grade application. This tech is not something to be afraid of. Better yet, some of these technologies provide us with a completely different way of thinking about the problem (like the shared storage solution).

2. Wiping code and rebuilding where needed

We learned not to be afraid to wipe out code and rebuild. Today, we look back at a small portion of a component which we left in C++ and realize that it was a mistake. We could have saved time and done a better job by simply rewriting it. Keeping the code of C components that were not originally written for multi-threaded operations instead of rewriting them to make them multi-threaded was eventually more expensive.

3. Keeping customer and end-user experience in mind

When we embarked on the Sisense on Linux deployment, it was very clear to us that we wanted to provide the same user experience in both the Windows and the Linux deployments.

A big reason for this was to ensure that we can use the carefully curated and built automated testing assets across both deployments. The automatic testing assets (various databases, different schemas, dashboards, validated results) were collected and built for the last couple of years. Keeping the same automatic testing assets was a top priority. The ability to test both deployments with the exact same assets is an important tool to ensure we were retaining data integrity between the two systems. This meant that, in certain areas, we choose not to change something on the front-end that we could have changed in order to ensure the end-user experience was not affected.

We also wanted to make the transition process from Windows to Linux (if asked for) to be quick and painless. To address this, we built a migration tool that allows our customers to move over all the work assets from Windows to Linux seamlessly so that they do not have to worry about rework.

4. Organization-wide focus and cross-company collaboration

A critical component of our success lay in cross-collaboration across R&D teams, and later with non-R&D teams, across the company. The Linux deployment is a completely new platform that touches every aspect of our organization and, at any given point, we had a significant number of Sisense developers contributing to it.

Additionally, this required changes outside of R&D.

Technical support teams needed to know how to debug issues and support customers using a completely new OS and new technology.
Pre-sales engineers needed to know how to successfully install and demo the new deployment to customers and needed to learn about the details of the new architecture. To facilitate the training of the tech teams, the teams not only subscribed to external courses but we also flew internal R&D trainers around the globe to share and educate the teams at various sisense locations.
Sales and marketing teams also needed to become familiar with cloud-technology and the benefits of the Cloud-Native Sisense on Linux deployment in order to convey these benefits to customers and prospects.

It was essential to garner buy-in across the organization with the full-support and prioritization coming from senior leadership. Without a vision and cross-organization goals, no project like this could come to fruition.

Summary

The Cloud-Native Sisense on Linux deployment marked a milestone in our journey as we became the only data and analytics platform with an advanced containerized microservices architecture that is purpose-built from the ground up with best-of-breed-technologies like Docker containers and Kubernetes orchestration that can be deployed on the cloud or on-premises. It provides the full Sisense platform including the Elastic Data Hub, which offers both live in-database connectivity to all major cloud databases, as well as Sisense’s proprietary In-Chip™ Performance Accelerator. The deployment fits seamlessly into DevOps processes and enables faster delivery, resiliency, and scalability.

We started this journey with a vision of building a true next-gen analytics platform that will lead the way in how organizations build large-scale analytic applications. We are proud of the platform being deployed with our customers today.

We successfully made this transition in little over a year and while we had some setbacks and difficulties (as with any project), the decisions around how to approach this project — like not shying away from rewriting components where needed — not only sped up the process but also allowed us to build a platform that can provide the most value to our customers in the cloud and the web based world we work in.

As we continue rolling out this new, full-stack Cloud-Native Sisense deployment, we are carefully working with teams across Sisense to make this a great experience for our customers and enable them to go from data to insights even faster in a highly-scalable and resilient environment.

CTA Linux 770x2501 770x250 How Sisense Engineered Its Cloud Native Linux Deployment From the Ground Up

Tags: Cloud | linux

Let’s block ads! (Why?)

Blog – Sisense

Time Spent Not Selling

Shorter Sales Cycles

Myth #1: Stability and resilience are key. Agile just adds risk.

Myth #2: Agile can’t provide the compliance, documentation, and approvals we need.

Myth #3: Waterfall works. There’s no need to reinvent the wheel.

Myth #4: A hybrid waterfall + agile approach is better.

Myth #6: Agile doesn’t work for greenfield implementations.

Myth #7: Agile doesn’t work with outsourced and offshore teams.

Conclusions

Your Current Information Technology Architecture is Expensive and Complex

You Don’t Have Sufficient IT Staffing

You Risk Breaking Your Business When You Introduce Change

Quantum Models

Other Date Displays

Closed Interval Models

ISO Half Open Interval Model

The OVERLAPS() Predicate

Keeping Contiguous Events

DATETIMEOFFSET

AT TIME ZONE

The Most Important Thing to take Home from this article

The true cost of spreadsheets

Stay streamlined

Maximize productivity

Report accurately

Improve employee satisfaction

Maintain compliance and efficiency

The great news is, you can do it!

The First Few Months

Taking Stock and Restarting the Project

The Right Technology for the Job

1. Sisense ElastiCube or Data Engine

2. ElastiCube Management Service and Query Service

3. Data connectors

Containerizing Microservices

A New Way of Doing Things with Shared Storage, Updated Monitoring & Logging

Learnings Along the Way

1. Embracing open source technologies

2. Wiping code and rebuilding where needed

3. Keeping customer and end-user experience in mind

4. Organization-wide focus and cross-company collaboration

Summary

Recent Posts

Categories

Archives