Monthly Archives: April 2016

SQL engines boost Hadoop query processing for big data users

TTlogo 379x201 SQL engines boost Hadoop query processing for big data users

When Premier Inc. offloaded some of its business intelligence data from a data warehouse appliance to a Hadoop cluster in mid-2014, the analytics and purchasing services company relied on MapReduce — Hadoop’s initial programming and processing environment — to pull in the supply chain information from healthcare providers. But batch-oriented MapReduce wasn’t seen as the right technology to power a Web-based BI dashboard application built for use by hospital purchasing managers and Premier’s own data analysts and supply chain account executives.

For that job, Premier turned to Impala, Hadoop vendor Cloudera Inc.’s SQL-on-Hadoop query engine. Based on SQL, the standard programming language for mainstream relational databases, Impala provided faster Hadoop query performance for the end users and a more familiar programming framework for Premier’s development team, according to Tom Palmer, a software engineering director who led the deployment of the Hadoop system at the Charlotte, N.C., company.

It was the same thing when Premier moved the processing of data on clinical outcomes and patient costs by physician from another data warehouse appliance to the Cloudera-based Hadoop cluster last September. After first trying to develop the required extract, transform and load (ETL) processes in MapReduce, the company switched to Impala before going into production.

We were able to develop a lot more, a lot faster, because [our ETL developers] were using the SQL syntax they were familiar with. Alfred Kosgeysenior technical architect at Premier Inc.

“The developers who created the ETL jobs, really, all they understood was SQL,” said Alfred Kosgey, a senior technical architect at Premier. “So, we were able to develop a lot more, a lot faster, because they were using the SQL syntax they were familiar with.”

Premier, which works with about 3,600 hospitals and 120,000 smaller healthcare providers in the U.S., is now also leaning on Impala for a new analytics platform that its data scientists will use to track industry norms on quality of care, length of stay and other performance metrics. The data scientists have been running queries created in the Python programming language on Macintosh systems, but Palmer said it can take “hours and hours” for the jobs to complete. That should be sped up, he added, by moving them to the Hadoop cluster and tapping Impala through Ibis, an open source technology that integrates Python and the SQL-on-Hadoop engine.

Premier is just one of a growing number of organizations looking to SQL-based Hadoop query tools to help simplify programming, and boost both ETL and analytics performance in their big data environments. The pairing of Hadoop and SQL also lets companies put all of their existing developers and data analysts who are versed in SQL to work on big data applications, reducing or even eliminating the need to invest in retraining or new hiring to build up MapReduce skills internally.

SQL-on-Hadoop options, issues

Prospective users have plenty of options to consider: Analysts from consultancies Gartner and Forrester Research have both counted more than a dozen SQL-on-Hadoop technologies, including a mix of open source software and commercial products. The top Hadoop vendors — Cloudera, Hortonworks Inc. and MapR Technologies Inc. — are all in the game, along with major IT providers, such as IBM, Microsoft, Oracle, Teradata and Hewlett Packard Enterprise. Jethro Data, Splice Machine Inc. and other startups are also pushing SQL-on-Hadoop offerings, as is Databricks Inc., the primary driving force behind the Apache Spark processing engine, which includes a Spark SQL module.

Most of the available tools are still relatively new and not yet fully mature. For example, many Hadoop query engines don’t support all of the functionality provided in relational SQL implementations. The fast pace of development on Hadoop and related technologies is helping to close that gap — but it also means organizations need to keep up with frequent new releases in order to take advantage of added SQL-on-Hadoop features. In addition, SQL itself can be too much for many business users, and even some BI and analytics professionals, to handle directly, often prompting companies to put SQL-on-Hadoop tools under the covers of self-service BI software or front-end Web user interfaces.

That all isn’t stopping early adopters, such as Premier and virtualization technology vendor VMware. The latter is building out a Hortonworks-based Hadoop cluster to power advanced analytics and predictive modeling applications after deploying a pilot system in early 2015. The cluster, which is due to go into production use this month, is being expanded from eight to 48 nodes, and will have a storage capacity of about 350 TB, said Joti Sidhu, IT director for enterprise BI applications at VMware.

In addition, the Palo Alto, Calif., company has deployed Pivotal HDB, a commercial version of the HAWQ open source SQL-on-Hadoop engine that’s sold by Pivotal Software Inc. — which VMware partly owns, along with its own parent, EMC. Sidhu said Pivotal HDB will primarily be used by a team of data scientists to run queries against a combination of marketing data, clickstream records from VMware’s website, and customer files that contain product log data and technical support information. “Those guys are working with huge data sets and essentially trying to find a needle in a haystack,” she said, adding that the SQL interface will let them do their work, without having to “learn or unlearn anything.”

Tight ties needed between tools

As the Hadoop ecosystem expands to include more and more technologies, most of which are being developed at a rapid pace, Sidhu said users have to ensure there’s tight integration between the underlying Hadoop platform and associated tools. There were some integration issues between Pivotal HDB and Hortonworks’ Hadoop distribution when VMware began working with them, she said, but the two vendors have brought the technologies closer together since then. To make sure that happens, “you have to work very closely with the product teams on the vendor side,” Sidhu advised.

Premier’s Palmer said the healthcare company also tries to deploy new versions of Cloudera’s Hadoop distribution and tools such as Impala within two to three months of their release. “Our attitude is that we’re going full speed ahead, so we can take advantage of the new functionality,” he said. “If you’re two releases behind, you’re not in good shape.”

Deploying a SQL-on-Hadoop query engine might require more processing horsepower in a Hadoop cluster, as well. For example, Premier added five compute nodes to its cluster when it moved the ETL processing for physician performance data to the Hadoop system, increasing the total number of nodes to 19. And Palmer said he expects the cluster to grow further as the company puts more applications on it.

But he added that the SQL-driven expansion of the cluster, which currently holds about 65 TB of data, has saved money overall by enabling Premier to completely replace one data warehouse appliance system and reduce its use of another one — both being higher-cost processing platforms than Hadoop. The new Hadoop math is simple, according to Palmer: “I like to say that the more we expand the cluster, the more money we save.”

Craig Stedman is executive editor of SearchBusinessAnalytics. Email him at, and follow us on Twitter: @BizAnalyticsTT.

Let’s block ads! (Why?)

SearchBusinessAnalytics: BI, CPM and analytics news, tips and resources



USD and the CSR: Part Two – Sessions

When it comes to mastering Unified Service Desk (USD), it’s important to remember CSR. CSR stands for Control-Session-Rule and is an approach to the most effective way to learn USD. In system design, quality relies on the vision and expertise of the team. To gain these strengths as your team

Read More

 USD and the CSR: Part Two – Sessions

Let’s block ads! (Why?)

PowerObjects- Bringing Focus to Dynamics CRM

The 10 biggest European tech stories this week

2015 08 28T104440Z 1 LYNXNPEB7R0GQ RTROPTP 3 FINLAND BUSINESS 780x519 The 10 biggest European tech stories this week

This week, tracked 11 technology M&A transactions and 63 funding deals (totalling €500 million, about $ 573 million) in Europe, Turkey and Israel.

Here’s an overview of the 10 biggest European tech news items for this week:

1) Rocket Internet’s ‘Global Fashion Group’ picked up €300 million in funding at a €1 billion valuation – roughly a third of the valuation it fetched less than a year ago. And boy did its stock price suffer.

2) Nokia announced plans to acquire Withings, a connected health product maker based in France, in a transaction that values the company at €170 million. Ben Rooney has a good take on the news.

3) According to Russian media reports, Pavel Durov – the founder of messaging app Telegram – met with Google CEO Sundar Pichai to discuss an acquisition valuing the startup at around $ 1 billion. Telegram has vehemently denied the rumours, though.

4) European mobile payment startups SumUp and Rocket Internet-backed Payleven are merging.

5) Slovenian company Bitstamp became the first Bitcoin exchange approved to operate in the EU (as in, all 28 member states).

6) EU leak reveals plans to rein in Google, Facebook, and encourage local alternatives.

7) SIRIN Labs, an Israeli-led venture with headquarters in London and initial R&D labs in both Tel Aviv and Lund (Sweden), has raised $ 72 million in funding to attempt to make some waves in the crowded global smartphone market. Also check out Geektime’s coverage of the news.

8) Other notable investments in Europe included France-based car rental marketplace startup Drivy (€31 million) and p2p-lender Lendix ($ 13.5 million), and German IoT tech company Tado ($ 23 million).

9) In Germany, VC firms Heilemann Ventures and Earlybird are merging and will set up new fund with a target size of €150 million, while Project A Ventures plans to announce next month that it’s closed a new fund of at least €120 million. Mo’ money for Berlin startups, in particular.

10) Germany is set to launch a new incentive scheme worth about 1 billion euros to get more consumers buying electric cars.

Bonus link: How Uber conquered London (The Guardian)

You can subscribe to’s newsletter here.

This post appeared first on

More: MobileBeat 2016 is focused on the paradigm shift from apps to AI, messaging, and chatbots. Don’t miss this opportunity: July 12 and 13 in San Francisco.

Let’s block ads! (Why?)

Big Data – VentureBeat

Is Dynamics CRM a Capital Expense or Operating Expense?

There are many ways to compare the pros and cons of Microsoft Dynamics CRM Online vs On-Premise. Tax ramifications are not usually considered, but they should be.

  • Microsoft Dynamics CRM Online is a monthly subscription cost. This is an operating expense.
  • Microsoft Dynamics CRM On-Premise is a large upfront cost with a small annual maintenance fee. This is a capital expense.

For example, you could have one big upfront payment of $ 1,000 per user license plus a 20% maintenance fee every year versus spending $ 65.00 for that license per month, forever.

In my experience larger companies tend to buy their Microsoft Dynamics CRM licenses, as a capital expense, versus renting it. If you list your CRM system as operating expense, over time it’s a very expensive option. If you capitalize the cost you just reduced your tax base.

Before purchasing Microsoft Dynamics CRM, or any new technology, you should look at the tax advantages plus government incentives for different types of investments.

So let your IT and Sales teams debate the value of the cloud. But make sure that your Finance team has the chance to give their vote on operating expense vs capital expense when choosing your Microsoft Dynamics CRM system.

If you are comparing Microsoft Dynamics CRM Online vs On Premise, and want honest advice, contact AbleBridge. 877-600-2253 or [email protected]

By AbleBridge, Microsoft Dynamics CRM Gold Partner,

728X90 7 625x77 Is Dynamics CRM a Capital Expense or Operating Expense?

Let’s block ads! (Why?)

CRM Software Blog

While RWNJs worry over toilets, actual criminal freed because attacking an unconscious victim is OK

 While RWNJs worry over toilets, actual criminal freed because attacking an unconscious victim is OK

watch out for tissue-filled bras

No one was trying to use a public toilet because gender, and yet a male took sexual advantage of a minor female. (see Ted Cruz and the fear for his daughters).

Young Conservatives apparently think that “Transgender people have always been around. Suddenly, during an election year bathrooms have become problematic and (a) political issue. Democrats are manipulative cultural terrorists and are trying to eliminate the existence of gender. “ Gender apparently is also stochastic, unlike sexuality.

However, in this case, the court found that an (unimpeachable) male apparently did not have sex with an unconscious female, despite, stuff like evidence (see rape kits ignored in other jurisdictions).

And the market for roofies will skyrocket in the Prairies since in some jurisdictions, Justice there is apparently blind drunk.

And RWNJs sign petitions protesting box store restroom policies that represent no clear or present danger. The American Family Association’s petition says that Target is openly inviting sexual predators to take advantage of their stores’ customers, asking “Where do you think predators are going to go?”

The place they seem to be going are either the inside of automobiles or going over the river and through the woods to grandmother’s house. Just go take the Sodom exit to Gomorrah. 

An Oklahoma court has stunned local prosecutors with a declaration that state law doesn’t criminalize oral sex with a victim who is completely unconscious. 

The case involved allegations that a 17-year-old boy assaulted a girl, 16, after volunteering to give her a ride home. The two had been drinking in a Tulsa park with a group of friends when it became clear that the girl was badly intoxicated. Witnesses recalled that she had to be carried into the defendant’s car. Another boy, who briefly rode in the car, recalled her coming in and out of consciousness.

The boy later brought the girl to her grandmother’s house. Still unconscious, the girl was taken to a hospital, where a test put her blood alcohol content above .34. She awoke as staff were conducting a sexual assault examination.

Tests would later confirm that the young man’s DNA was found on the back of her leg and around her mouth. The boy claimed to investigators that the girl had consented to performing oral sex. The girl said she didn’t have any memories after leaving the park. Tulsa County prosecutors charged the young man with forcible oral sodomy.

But the trial judge dismissed the case. And the appeals court ruling, on 24 March, affirmed that prosecutors could not apply the law to a victim who was incapacitated by alcohol.

“Forcible sodomy cannot occur where a victim is so intoxicated as to be completely unconscious at the time of the sexual act of oral copulation,” the decision read. Its reasoning, the court said, was that the statute listed several circumstances that constitute force, and yet was silent on incapacitation due to the victim drinking alcohol. “We will not, in order to justify prosecution of a person for an offense, enlarge a statute beyond the fair meaning of its language.”…

Let’s block ads! (Why?)


How To Eliminate Cultural Distractions For Greater Business Success

The September issue of the Harvard Business Review features a cover story on design thinking’s coming of age. We have been applying design thinking within SAP for the past 10 years, and I’ve witnessed the growth of this human-centered approach to innovation first hand.

Design thinking is, as the HBR piece points out, “the best tool we have for … developing a responsive, flexible organizational culture.”

This means businesses are doing more to learn about their customers by interacting directly with them. We’re seeing this change in our work on — a community of design thinking champions and “disruptors” from across industries.

Meanwhile, technology is making it possible to know exponentially more about a customer. Businesses can now make increasingly accurate predictions about customers’ needs well into the future. The businesses best able to access and pull insights from this growing volume of data will win. That requires a fundamental change for our own industry; it necessitates a digital transformation.

So, how do we design this digital transformation?

It starts with the customer and an application of design thinking throughout an organization – blending business, technology and human values to generate innovation. Business is already incorporating design thinking, as the HBR cover story shows. We in technology need to do the same.

SCN+SY How To Eliminate Cultural Distractions For Greater Business Success

Design thinking plays an important role because it helps articulate what the end customer’s experience is going to be like. It helps focus all aspects of the business on understanding and articulating that future experience.

Once an organization is able to do that, the insights from that consumer experience need to be drawn down into the business, with the central question becoming: What does this future customer experience mean for us as an organization? What barriers do we need to remove? Do we need to organize ourselves differently? Does our process need to change – if it does, how? What kind of new technology do we need?

Then an organization must look carefully at roles within itself. What does this knowledge of the end customer’s future experience mean for an individual in human resources, for example, or finance? Those roles can then be viewed as end experiences unto themselves, with organizations applying design thinking to learn about the needs inherent to those roles. They can then change roles to better meet the end customer’s future needs. This end customer-centered approach is what drives change.

This also means design thinking is more important than ever for IT organizations.

We, in the IT industry, have been charged with being responsive to business, using technology to solve the problems business presents. Unfortunately, business sometimes views IT as the organization keeping the lights on. If we make the analogy of a store: business is responsible for the front office, focused on growing the business where consumers directly interact with products and marketing; while the perception is that IT focuses on the back office, keeping servers running and the distribution system humming. The key is to have business and IT align to meet the needs of the front office together.

Remember what I said about the growing availability of consumer data? The business best able to access and learn from that data will win. Those of us in IT organizations have the technology to make that win possible, but the way we are seen and our very nature needs to change if we want to remain relevant to business and participate in crafting the winning strategy.

We need to become more front office and less back office, proving to business that we are innovation partners in technology.

This means, in order to communicate with businesses today, we need to take a design thinking approach. We in IT need to show we have an understanding of the end consumer’s needs and experience, and we must align that knowledge and understanding with technological solutions. When this works — when the front office and back office come together in this way — it can lead to solutions that a company could otherwise never have realized.

There’s different qualities, of course, between front office and back office requirements. The back office is the foundation of a company and requires robustness, stability, and reliability. The front office, on the other hand, moves much more quickly. It is always changing with new product offerings and marketing campaigns. Technology must also show agility, flexibility, and speed. The business needs both functions to survive. This is a challenge for IT organizations, but it is not an impossible shift for us to make.

Here’s the breakdown of our challenge.

1. We need to better understand the real needs of the business.

This means learning more about the experience and needs of the end customer and then translating that information into technological solutions.

2. We need to be involved in more of the strategic discussions of the business.

Use the regular invitations to meetings with business as an opportunity to surface the deeper learning about the end consumer and the technology solutions that business may otherwise not know to ask for or how to implement.

The IT industry overall may not have a track record of operating in this way, but if we are not involved in the strategic direction of companies and shedding light on the future path, we risk not being considered innovation partners for the business.

We must collaborate with business, understand the strategic direction and highlight the technical challenges and opportunities. When we do, IT will become a hybrid organization – able to maintain the back office while capitalizing on the front office’s growing technical needs. We will highlight solutions that business could otherwise have missed, ushering in a digital transformation.

Digital transformation goes beyond just technology; it requires a mindset. See What It Really Means To Be A Digital Organization.

This story originally appeared on SAP Business Trends.

Top image via Shutterstock


Let’s block ads! (Why?)

Digitalist Magazine

Dashboard Makeover: learn how to make your data shine

Good analysis depends on good data, but it’s up to the look and design of your dashboard to properly deliver insights to your audience. Is the visual design of your work as good as the underlying data, or does it distract from your message?

Ask the experts at Microsoft and find out! We’re holding a Dashboard Makeover to review, revise, and improve entries from our Community. Attend the live webinar on May 12, 2016 to see how our Power BI pros turn dashboards from standard to shiny, sharing insights more efficiently and making a bigger impact on your audience.

Enter Your Dashboard or Report for a Makeover

We want your dashboard to instantly impress its audience, so the Power BI team is offering the chance for a Dashboard Makeover.

8ae78631 9850 4f0f 9984 afd3e0f19b7f Dashboard Makeover: learn how to make your data shineSubmit your entry by May 5, 2016, and we’ll select one or more to be reviewed and remodeled during a live webinar with Microsoft Principal Program Manager and Power BI dashboard expert Marc Reguera. The selections will be chosen by Marc himself to demonstrate potential in the areas of readability, formatting, visual usage, and functionality.

The owner of the selected entry(s) will win a Power BI Prize Pack, as well as a newly renovated dashboard. 

Watch the Dashboard Makeover Webinar

If you want to learn more about dashboard design, with or without a submission of your own, register now for the Dashboard Makeover webinar! You’ll watch Marc’s dashboard critique in real-time and pick up some useful tips on creating a professionally designed report. The webinar is free, and attendees will have a chance to win one of our new Power BI t-shirts.

For all of the entry rules and webinar details, visit the Dashboard Makeover forum on our Community.

Let’s block ads! (Why?)

Microsoft Power BI Blog | Microsoft Power BI

Late Night Political Humor

“The big New York primary, which happens next Tuesday, is looking pretty good for Hillary Clinton. In fact, website FiveThirtyEight says Hillary has a 99 percent chance of winning the primary for New York. When he heard, Bernie Sanders said, ‘My God, I’ve become part of the 1 percent!’” – Jimmy Fallon

“Bernie Sanders this morning joined the Verizon workers picket line here in New York. It’s a perfect match, because Bernie always talks like he’s getting bad reception.” – Seth Meyers

“Bernie Sanders today received his first senatorial endorsement from Oregon Sen. Jeff Merkley. Or as he’ll be known under President Hillary Clinton, ‘Ambassador to North Korea Jeff Merkley.’” – Seth Meyers

“Trump’s family was also at the town hall, and Trump’s daughter Ivanka was asked if the election is straining her friendship with Chelsea Clinton. Which means we have officially begun the presidential campaign of 2032, everybody!” – Jimmy Fallon

“Donald Trump’s son Eric said last night that his father is his ‘best friend in the entire world.’ Said Donald, ‘Right back at ya, Jeff.’” – Seth Meyers

“Last night, CNN hosted a town hall with Republican front-runner Donald Trump, and at one point he ‘complained that the rules of the election are stacked against him ‘by the establishment. You gotta give it to Trump. He’s the only man who could inherit millions of dollars, have his name on buildings, and still go, ‘Life is totally unfair!’” – Jimmy Fallon

“Some prominent Republican congressmen are saying they might not even go to the convention, which is in Cleveland this summer. Not because it might get crazy — they’re saying they can’t go because they have work to do. This summer. These are congressmen. All of a sudden they’ve got work to do?” – Jimmy Fallon

share save 256 24 Late Night Political Humor

Let’s block ads! (Why?)

Political Irony

DeepMind AI group moves from Torch framework to Google’s own TensorFlow

Google GDC 2016 Novet 780x551 DeepMind AI group moves from Torch framework to Google’s own TensorFlow

Google’s DeepMind artificial intelligence (AI) research group today announced that for all future research it will use TensorFlow, a machine learning library that Google open-sourced last year, instead of Torch, an older framework.

The move suggests that some of Google’s brightest AI minds are convinced of the promise of Google’s own open source software; TensorFlow is now good enough for DeepMind.

“We believe that TensorFlow will enable us to execute our ambitious research goals at much larger scale and an even faster pace, providing us with a unique opportunity to further accelerate our research programme,” Koray Kavukcuoglu, a research scientist at Google DeepMind and one of Torch’s core contributors, wrote in a blog post.

This is important because of DeepMind’s considerable capabilities — earlier this year its AlphaGo AI player of the ancient Chinese board game Go beat top-ranked Go player Lee Sedol.

To be sure, DeepMind is not Google’s only AI research unit. Google also has the larger Google Brain team.

In a letter to Alphabet shareholders yesterday, Google chief executive Sundar Pichai played up the importance of AI. “We’ve been building the best AI team and tools for years, and recent breakthroughs will allow us
to do even more,” Pichai wrote.

There are several other open-source deep learning frameworks to choose from, but it would only be right for Google’s internal groups to gradually align with its own open source tooling.

“Our transition to TensorFlow represents a new chapter, and I feel very excited about the prospect of DeepMind contributing heavily to another great open source machine learning platform that everyone can use to advance the state-of-the-art,” Kavukcuoglu wrote.

Google said earlier this month that TensorFlow could now support training across multiple machines, not just one.

More: MobileBeat 2016 is focused squarely on the paradigm shift from apps to AI, messaging, and chatbots. Don’t miss this opportunity to be on the cusp of the next mobile transformation. July 11 and 12, San Francisco.

Let’s block ads! (Why?)

Big Data – VentureBeat