Category Archives: BI News and Info

Released: Microsoft Kerberos Configuration Manager for SQL Server 4.0

We are pleased to announce the latest generally-available (GA) of Microsoft Kerberos Configuration Manager for SQL Server.

Get it here:Download Microsoft Kerberos Configuration Manager for SQL Server

Why Kerberos?

Kerberos authentication provides a highly secure method to authenticate client and server entities (security principals) on a network. To use Kerberos authentication with SQL Server, a Service Principal Name (SPN) must be registered with Active Directory, which plays the role of the Key Distribution Center in a Windows domain. In addition, many customers also enable delegation for multi-tier applications using SQL Server. In such a setup, it may be difficult to troubleshoot the connectivity problems with SQL Server when Kerberos authentication fails.

Here are some additional reading materials for your reference.

Why use this tool?

The Kerberos Configuration Manager for SQL Server is a diagnostic tool that helps troubleshoot Kerberos related connectivity issues with SQL Server, SQL Server Reporting Services, and SQL Server Analysis Services. It can perform the following functions:

  • Gather information on OS and Microsoft SQL Server instances installed on a server.
  • Report on all SPN and delegation configurations and Always On Availability Group Listeners installed on a server.
  • Identify potential problems in SPNs and delegations.
  • Fix potential SPN problems.

This release (v4.0) adds support for Always On Availability Group Listeners.

Notes
  • Microsoft Kerberos Configuration Manager for SQL Server requires a user with permission to connect to the WMI service on any machine its connecting to. For more information, refer to Securing a Remote WMI Connection.
  • For Always On Availability Group Listeners discovery, run this tool from the owner node.
  • Also, if needed for troubleshooting, the Kerberos Configuration Manager for SQL Server creates a log file in %AppData%\Microsoft\KerberosConfigMgr.

Let’s block ads! (Why?)

SQL Server Release Services

Why Digital Transformation Should Focus On Growth, Not Disruption

In the tech world in 2017, several trends emerged as signals amid the noise, signifying much larger changes to come.

As we noted in last year’s More Than Noise list, things are changing—and the changes are occurring in ways that don’t necessarily fit into the prevailing narrative.

While many of 2017’s signals have a dark tint to them, perhaps reflecting the times we live in, we have sought out some rays of light to illuminate the way forward. The following signals differ considerably, but understanding them can help guide businesses in the right direction for 2018 and beyond.

SAP Q417 DigitalDoubles Feature1 Image2 1024x572 Why Digital Transformation Should Focus On Growth, Not Disruption

When a team of psychologists, linguists, and software engineers created Woebot, an AI chatbot that helps people learn cognitive behavioral therapy techniques for managing mental health issues like anxiety and depression, they did something unusual, at least when it comes to chatbots: they submitted it for peer review.

Stanford University researchers recruited a sample group of 70 college-age participants on social media to take part in a randomized control study of Woebot. The researchers found that their creation was useful for improving anxiety and depression symptoms. A study of the user interaction with the bot was submitted for peer review and published in the Journal of Medical Internet Research Mental Health in June 2017.

While Woebot may not revolutionize the field of psychology, it could change the way we view AI development. Well-known figures such as Elon Musk and Bill Gates have expressed concerns that artificial intelligence is essentially ungovernable. Peer review, such as with the Stanford study, is one way to approach this challenge and figure out how to properly evaluate and find a place for these software programs.

The healthcare community could be onto something. We’ve already seen instances where AI chatbots have spun out of control, such as when internet trolls trained Microsoft’s Tay to become a hate-spewing misanthrope. Bots are only as good as their design; making sure they stay on message and don’t act in unexpected ways is crucial.

SAP Q417 DigitalDoubles Feature1 Image3 Why Digital Transformation Should Focus On Growth, Not DisruptionThis is especially true in healthcare. When chatbots are offering therapeutic services, they must be properly designed, vetted, and tested to maintain patient safety.

It may be prudent to apply the same level of caution to a business setting. By treating chatbots as if they’re akin to medicine or drugs, we have a model for thorough vetting that, while not perfect, is generally effective and time tested.

It may seem like overkill to think of chatbots that manage pizza orders or help resolve parking tickets as potential health threats. But it’s already clear that AI can have unintended side effects that could extend far beyond Tay’s loathsome behavior.

For example, in July, Facebook shut down an experiment where it challenged two AIs to negotiate with each other over a trade. When the experiment began, the two chatbots quickly went rogue, developing linguistic shortcuts to reduce negotiating time and leaving their creators unable to understand what they were saying.

The implications are chilling. Do we want AIs interacting in a secret language because designers didn’t fully understand what they were designing?

In this context, the healthcare community’s conservative approach doesn’t seem so farfetched. Woebot could ultimately become an example of the kind of oversight that’s needed for all AIs.

Meanwhile, it’s clear that chatbots have great potential in healthcare—not just for treating mental health issues but for helping patients understand symptoms, build treatment regimens, and more. They could also help unclog barriers to healthcare, which is plagued worldwide by high prices, long wait times, and other challenges. While they are not a substitute for actual humans, chatbots can be used by anyone with a computer or smartphone, 24 hours a day, seven days a week, regardless of financial status.

Finding the right governance for AI development won’t happen overnight. But peer review, extensive internal quality analysis, and other processes will go a long way to ensuring bots function as expected. Otherwise, companies and their customers could pay a big price.

SAP Q417 DigitalDoubles Feature1 Image4 1024x572 Why Digital Transformation Should Focus On Growth, Not Disruption

Elon Musk is an expert at dominating the news cycle with his sci-fi premonitions about space travel and high-speed hyperloops. However, he captured media attention in Australia in April 2017 for something much more down to earth: how to deal with blackouts and power outages.

In 2016, a massive blackout hit the state of South Australia following a storm. Although power was restored quickly in Adelaide, the capital, people in the wide stretches of arid desert that surround it spent days waiting for the power to return. That hit South Australia’s wine and livestock industries especially hard.

South Australia’s electrical grid currently gets more than half of its energy from wind and solar, with coal and gas plants acting as backups for when the sun hides or the wind doesn’t blow, according to ABC News Australia. But this network is vulnerable to sudden loss of generation—which is exactly what happened in the storm that caused the 2016 blackout, when tornadoes ripped through some key transmission lines. Getting the system back on stable footing has been an issue ever since.

Displaying his usual talent for showmanship, Musk stepped in and promised to build the world’s largest battery to store backup energy for the network—and he pledged to complete it within 100 days of signing the contract or the battery would be free. Pen met paper with South Australia and French utility Neoen in September. As of press time in November, construction was underway.

For South Australia, the Tesla deal offers an easy and secure way to store renewable energy. Tesla’s 129 MWh battery will be the most powerful battery system in the world by 60% once completed, according to Gizmodo. The battery, which is stationed at a wind farm, will cover temporary drops in wind power and kick in to help conventional gas and coal plants balance generation with demand across the network. South Australian citizens and politicians largely support the project, which Tesla claims will be able to power 30,000 homes.

Until Musk made his bold promise, batteries did not figure much in renewable energy networks, mostly because they just aren’t that good. They have limited charges, are difficult to build, and are difficult to manage. Utilities also worry about relying on the same lithium-ion battery technology as cellphone makers like Samsung, whose Galaxy Note 7 had to be recalled in 2016 after some defective batteries burst into flames, according to CNET.

SAP Q417 DigitalDoubles Feature1 Image5 Why Digital Transformation Should Focus On Growth, Not DisruptionHowever, when made right, the batteries are safe. It’s just that they’ve traditionally been too expensive for large-scale uses such as renewable power storage. But battery innovations such as Tesla’s could radically change how we power the economy. According to a study that appeared this year in Nature, the continued drop in the cost of battery storage has made renewable energy price-competitive with traditional fossil fuels.

This is a massive shift. Or, as David Roberts of news site Vox puts it, “Batteries are soon going to disrupt power markets at all scales.” Furthermore, if the cost of batteries continues to drop, supply chains could experience radical energy cost savings. This could disrupt energy utilities, manufacturing, transportation, and construction, to name just a few, and create many opportunities while changing established business models. (For more on how renewable energy will affect business, read the feature “Tick Tock” in this issue.)

Battery research and development has become big business. Thanks to electric cars and powerful smartphones, there has been incredible pressure to make more powerful batteries that last longer between charges.

The proof of this is in the R&D funding pudding. A Brookings Institution report notes that both the Chinese and U.S. governments offer generous subsidies for lithium-ion battery advancement. Automakers such as Daimler and BMW have established divisions marketing residential and commercial energy storage products. Boeing, Airbus, Rolls-Royce, and General Electric are all experimenting with various electric propulsion systems for aircraft—which means that hybrid airplanes are also a possibility.

Meanwhile, governments around the world are accelerating battery research investment by banning internal combustion vehicles. Britain, France, India, and Norway are seeking to go all electric as early as 2025 and by 2040 at the latest.

In the meantime, expect huge investment and new battery innovation from interested parties across industries that all share a stake in the outcome. This past September, for example, Volkswagen announced a €50 billion research investment in batteries to help bring 300 electric vehicle models to market by 2030.

SAP Q417 DigitalDoubles Feature1 Image6 1024x572 Why Digital Transformation Should Focus On Growth, Not Disruption

At first, it sounds like a narrative device from a science fiction novel or a particularly bad urban legend.

Powerful cameras in several Chinese cities capture photographs of jaywalkers as they cross the street and, several minutes later, display their photograph, name, and home address on a large screen posted at the intersection. Several days later, a summons appears in the offender’s mailbox demanding payment of a fine or fulfillment of community service.

As Orwellian as it seems, this technology is very real for residents of Jinan and several other Chinese cities. According to a Xinhua interview with Li Yong of the Jinan traffic police, “Since the new technology has been adopted, the cases of jaywalking have been reduced from 200 to 20 each day at the major intersection of Jingshi and Shungeng roads.”

The sophisticated cameras and facial recognition systems already used in China—and their near–real-time public shaming—are an example of how machine learning, mobile phone surveillance, and internet activity tracking are being used to censor and control populations. Most worryingly, the prospect of real-time surveillance makes running surveillance states such as the former East Germany and current North Korea much more financially efficient.

According to a 2015 discussion paper by the Institute for the Study of Labor, a German research center, by the 1980s almost 0.5% of the East German population was directly employed by the Stasi, the country’s state security service and secret police—1 for every 166 citizens. An additional 1.1% of the population (1 for every 66 citizens) were working as unofficial informers, which represented a massive economic drain. Automated, real-time, algorithm-driven monitoring could potentially drive the cost of controlling the population down substantially in police states—and elsewhere.

We could see a radical new era of censorship that is much more manipulative than anything that has come before. Previously, dissidents were identified when investigators manually combed through photos, read writings, or listened in on phone calls. Real-time algorithmic monitoring means that acts of perceived defiance can be identified and deleted in the moment and their perpetrators marked for swift judgment before they can make an impression on others.

SAP Q417 DigitalDoubles Feature1 Image7 Why Digital Transformation Should Focus On Growth, Not DisruptionBusinesses need to be aware of the wider trend toward real-time, automated censorship and how it might be used in both commercial and governmental settings. These tools can easily be used in countries with unstable political dynamics and could become a real concern for businesses that operate across borders. Businesses must learn to educate and protect employees when technology can censor and punish in real time.

Indeed, the technologies used for this kind of repression could be easily adapted from those that have already been developed for businesses. For instance, both Facebook and Google use near–real-time facial identification algorithms that automatically identify people in images uploaded by users—which helps the companies build out their social graphs and target users with profitable advertisements. Automated algorithms also flag Facebook posts that potentially violate the company’s terms of service.

China is already using these technologies to control its own people in ways that are largely hidden to outsiders.

According to a report by the University of Toronto’s Citizen Lab, the popular Chinese social network WeChat operates under a policy its authors call “One App, Two Systems.” Users with Chinese phone numbers are subjected to dynamic keyword censorship that changes depending on current events and whether a user is in a private chat or in a group. Depending on the political winds, users are blocked from accessing a range of websites that report critically on China through WeChat’s internal browser. Non-Chinese users, however, are not subject to any of these restrictions.

The censorship is also designed to be invisible. Messages are blocked without any user notification, and China has intermittently blocked WhatsApp and other foreign social networks. As a result, Chinese users are steered toward national social networks, which are more compliant with government pressure.

China’s policies play into a larger global trend: the nationalization of the internet. China, Russia, the European Union, and the United States have all adopted different approaches to censorship, user privacy, and surveillance. Although there are social networks such as WeChat or Russia’s VKontakte that are popular in primarily one country, nationalizing the internet challenges users of multinational services such as Facebook and YouTube. These different approaches, which impact everything from data safe harbor laws to legal consequences for posting inflammatory material, have implications for businesses working in multiple countries, as well.

For instance, Twitter is legally obligated to hide Nazi and neo-fascist imagery and some tweets in Germany and France—but not elsewhere. YouTube was officially banned in Turkey for two years because of videos a Turkish court deemed “insulting to the memory of Mustafa Kemal Atatürk,” father of modern Turkey. In Russia, Google must keep Russian users’ personal data on servers located inside Russia to comply with government policy.

While China is a pioneer in the field of instant censorship, tech companies in the United States are matching China’s progress, which could potentially have a chilling effect on democracy. In 2016, Apple applied for a patent on technology that censors audio streams in real time—automating the previously manual process of censoring curse words in streaming audio.

SAP Q417 DigitalDoubles Feature1 Image8 1024x572 Why Digital Transformation Should Focus On Growth, Not Disruption

In March, after U.S. President Donald Trump told Fox News, “I think maybe I wouldn’t be [president] if it wasn’t for Twitter,” Twitter founder Evan “Ev” Williams did something highly unusual for the creator of a massive social network.

He apologized.

Speaking with David Streitfeld of The New York Times, Williams said, “It’s a very bad thing, Twitter’s role in that. If it’s true that he wouldn’t be president if it weren’t for Twitter, then yeah, I’m sorry.”

Entrepreneurs tend to be very proud of their innovations. Williams, however, offers a far more ambivalent response to his creation’s success. Much of the 2016 presidential election’s rancor was fueled by Twitter, and the instant gratification of Twitter attracts trolls, bullies, and bigots just as easily as it attracts politicians, celebrities, comedians, and sports fans.

Services such as Twitter, Facebook, YouTube, and Instagram are designed through a mix of look and feel, algorithmic wizardry, and psychological techniques to hang on to users for as long as possible—which helps the services sell more advertisements and make more money. Toxic political discourse and online harassment are unintended side effects of the economic-driven urge to keep users engaged no matter what.

Keeping users’ eyeballs on their screens requires endless hours of multivariate testing, user research, and algorithm refinement. For instance, Casey Newton of tech publication The Verge notes that Google Brain, Google’s AI division, plays a key part in generating YouTube’s video recommendations.

According to Jim McFadden, the technical lead for YouTube recommendations, “Before, if I watch this video from a comedian, our recommendations were pretty good at saying, here’s another one just like it,” he told Newton. “But the Google Brain model figures out other comedians who are similar but not exactly the same—even more adjacent relationships. It’s able to see patterns that are less obvious.”

SAP Q417 DigitalDoubles Feature1 Image9 Why Digital Transformation Should Focus On Growth, Not DisruptionA never-ending flow of content that is interesting without being repetitive is harder to resist. With users glued to online services, addiction and other behavioral problems occur to an unhealthy degree. According to a 2016 poll by nonprofit research company Common Sense Media, 50% of American teenagers believe they are addicted to their smartphones.

This pattern is extending into the workplace. Seventy-five percent of companies told research company Harris Poll in 2016 that two or more hours a day are lost in productivity because employees are distracted. The number one reason? Cellphones and texting, according to 55% of those companies surveyed. Another 41% pointed to the internet.

Tristan Harris, a former design ethicist at Google, argues that many product designers for online services try to exploit psychological vulnerabilities in a bid to keep users engaged for longer periods. Harris refers to an iPhone as “a slot machine in my pocket” and argues that user interface (UI) and user experience (UX) designers need to adopt something akin to a Hippocratic Oath to stop exploiting users’ psychological vulnerabilities.

In fact, there is an entire school of study devoted to “dark UX”—small design tweaks to increase profits. These can be as innocuous as a “Buy Now” button in a visually pleasing color or as controversial as when Facebook tweaked its algorithm in 2012 to show a randomly selected group of almost 700,000 users (who had not given their permission) newsfeeds that skewed more positive to some users and more negative to others to gauge the impact on their respective emotional states, according to an article in Wired.

As computers, smartphones, and televisions come ever closer to convergence, these issues matter increasingly to businesses. Some of the universal side effects of addiction are lost productivity at work and poor health. Businesses should offer training and help for employees who can’t stop checking their smartphones.

Mindfulness-centered mobile apps such as Headspace, Calm, and Forest offer one way to break the habit. Users can also choose to break internet addiction by going for a walk, turning their computers off, or using tools like StayFocusd or Freedom to block addictive websites or apps.

Most importantly, companies in the business of creating tech products need to design software and hardware that discourages addictive behavior. This means avoiding bad designs that emphasize engagement metrics over human health. A world of advertising preroll showing up on smart refrigerator touchscreens at 2 a.m. benefits no one.

According to a 2014 study in Cyberpsychology, Behavior and Social Networking, approximately 6% of the world’s population suffers from internet addiction to one degree or another. As more users in emerging economies gain access to cheap data, smartphones, and laptops, that percentage will only increase. For businesses, getting a head start on stopping internet addiction will make employees happier and more productive. D!


About the Authors

Maurizio Cattaneo is Director, Delivery Execution, Energy, and Natural Resources, at SAP.

David Delaney is Global Vice President and Chief Medical Officer, SAP Health.

Volker Hildebrand is Global Vice President for SAP Hybris solutions.

Neal Ungerleider is a Los Angeles-based technology journalist and consultant.


Read more thought provoking articles in the latest issue of the Digitalist Magazine, Executive Quarterly.

Comments

Let’s block ads! (Why?)

Digitalist Magazine

Oracle Big Data SQL 3.2 is Now Available

Big Data SQL 3.2 has been released and is now available for download on edelivery.  This new release has many exciting new features – with a focus on simpler install and configuration, support for new data sources, enhanced security and improved performance.

Big Data SQL has expanded its data source support to now include querying data streams – specifically Kafka topics:

This enables streaming data to be joined with dimensions and facts in Oracle Database or HDFS.  It’s never been easier to combine data from streams, Hadoop and Oracle Database.

New security capabilities enable Big Data SQL to automatically leverage underlying authorization rules on source data (i.e. ACLs on HDFS data) and then augment that with Oracle’s advanced security policies.  In addition, to prevent impersonation, Oracle Database servers now authenticate against Big Data SQL Server cells. Finally, secure Big Data SQL installations have become much easier to set up; Kerberos ticket renewals are now automatically configured.

There has been significant performance improvements as well.  Oracle now provides its own optimized Parquet driver which delivers a significant performance boost – both in terms of speed and the ability to query many columns.  Support for CLOBs is also now available – which facilitates efficient processing of large JSON and XML data documents.

Finally, there has been significant enhancements to the out-of-box experience.  The installation process has been simplified, streamlined and made much more robust.

Let’s block ads! (Why?)

Oracle Blogs | Oracle The Data Warehouse Insider Blog

SeriesCoefficient is broken in Mathematica 11.1, but works in 11.0

 SeriesCoefficient is broken in Mathematica 11.1, but works in 11.0

Consider the following implementation of the complex square root:

f[z_]:=Sqrt[(z - I)/(z + I)]*(z + I);

This implementation has branch points at $ \lambda=\pm i$ and a (vertical) branch cut connecting them.

Then

g[z_]:=Sinc[f[z]];

(recalling $ \mathrm{sinc}(x)=\sin(x)/x$ ) has no branch cut and it is analytic on the entire complex plane, and admits power series expansions at $ \lambda=\pm i$ . Indeed, using Mathematica 11.0.0 (Mac OS 10.10.5) gives:

Series[Sinc[rhofun[z]], {z, I, 4}]

$ 1-\frac{1}{3} i (z-i)-\frac{1}{5} (z-i)^2+\frac{11}{315} i (z-i)^3+\frac{61
(z-i)^4}{5670}+O\left((z-i)^5\right)$

and

SeriesCoefficient[Sinc[rhofun[z]], {z, I, 4}]

gives $ \frac{61}{5670}$ .

Now, using Mathematica 11.1.1 (both on Mac OS 10.12 Sierra and Linux Ubuntu 16 LTS)

Series[Sinc[rhofun[z]], {z, I, 4}]

returns

Series[Sinc[rhofun[z]], {z, I, 4}]

and

SeriesCoefficient[Sinc[rhofun[z]], {z, I, 4}]

returns

SeriesCoefficient[Sinc[rhofun[z]], {z, I, 4}].

So neither of these stock functions work in properly in Mathematica 11.1.1. Does anyone know what is going on? Will this be fixed? They worked properly even in Mathematica 9 and also in Mathematica 11.0.0

Besides any information, I’d also appreciate if anyone has a workaround for this.

Let’s block ads! (Why?)

Recent Questions – Mathematica Stack Exchange

Big Data SQL Quick Start. Big Data SQL over Kafka – Part 23

G Data SQL 3.2 version brings a few interesting features. Among those features, one of the most interesting is the ability to read Kafka. Before drilling down into details, I’d like to explain in the nutshell what Kafka is.

What is Kafka?

The full scope of the information about Kafka you may find here, but in the nutshell, it’s distributed fault tolerant message system. It allows you to connect many systems in an organized fashion. Instead, connect each system peer to peer:

you may land all your messages company wide on one system and consume it from there, like this:

Kafka is kind of Data Hub system, where you land the messages and serve it after.

More technical details.

I’d like to introduce a few key Kafka’s terms.

1) Kafka Broker. This is Kafka service, which you run on each server and which operates all read and write request

2) Kafka Producer. The process which writes data in Kafka

3) Kafka Consumer. The process, which reads data from Kafka.

4) Message. The name describes itself, I just want to add that messages have key and value. In comparison to NoSQL databases key Kafka’s key is not indexed. It has application purposes (you may put some application logic in Key) and administrative purposes (each message with the same key goes to the same partition).

5) Topic. Set of the messages organized into topics. Database guys would compare it with a table.

6) Partition. It’s a good practice to divide the topic into partitions for performance and maintenance purposes. Messages within the same key go to the same partition. If a key is absent, messages are distributing in round – robin fashion.

7) Offset. The offset is the position of each message in the topic. The offset is indexed and it allows you quickly access your particular message.

When do you delete data?

One of the basic Kafka concepts is that of retention – Kafka does not keep data forever, nor does it wait for all consumers to read a message before deleting a message. Instead, the Kafka administrator configures a retention period for each topic – either amount of time for which to store messages before deleting them or how much data to store older messages are purged. This two parameters control this: log.retention.ms and log.retention.bytes.

The amount of data to retain in the log for each topic-partition. This is the limit per partition: multiply by the number of partitions to get the total data retained for the topic. 

How to query Kafka data with Big Data SQL?

for query the Kafka data you need to create hive table first. let me show an ent-to-end example. I do have a JSON file:

$   cat web_clicks.json
{ click_date: "38041", click_time: "67786", date: "2004-02-26", am_pm: "PM", shift: "second", sub_shift: "evening", item_sk: "396439", web_page: "646"}
{ click_date: "38041", click_time: "41831", date: "2004-02-26", am_pm: "AM", shift: "first", sub_shift: "morning", item_sk: "90714", web_page: "804"}
{ click_date: "38041", click_time: "60334", date: "2004-02-26", am_pm: "PM", shift: "second", sub_shift: "afternoon", item_sk: "151944", web_page: "867"}
{ click_date: "38041", click_time: "53225", date: "2004-02-26", am_pm: "PM", shift: "first", sub_shift: "afternoon", item_sk: "175796", web_page: "563"}
{ click_date: "38041", click_time: "47515", date: "2004-02-26", am_pm: "PM", shift: "first", sub_shift: "afternoon", item_sk: "186943", web_page: "777"}
{ click_date: "38041", click_time: "73633", date: "2004-02-26", am_pm: "PM", shift: "second", sub_shift: "evening", item_sk: "118004", web_page: "647"}
{ click_date: "38041", click_time: "43133", date: "2004-02-26", am_pm: "AM", shift: "first", sub_shift: "morning", item_sk: "148210", web_page: "930"}
{ click_date: "38041", click_time: "80675", date: "2004-02-26", am_pm: "PM", shift: "second", sub_shift: "evening", item_sk: "380306", web_page: "484"}
{ click_date: "38041", click_time: "21847", date: "2004-02-26", am_pm: "AM", shift: "third", sub_shift: "morning", item_sk: "55425", web_page: "95"}
{ click_date: "38041", click_time: "35131", date: "2004-02-26", am_pm: "AM", shift: "first", sub_shift: "morning", item_sk: "185071", web_page: "118"}

and I’m going to load it into Kafka with standard Kafka tool “kafka-console-producer”:

$   cat web_clicks.json|kafka-console-producer --broker-list bds2:9092,bds3:9092,bds4:9092,bds5:9092,bds6:9092 --topic json_clickstream

for a check that messages have appeared in the topic you may use the following command:

$   kafka-console-consumer --zookeeper bds1:2181,bds2:2181,bds3:2181 --topic json_clickstream --from-beginning

after I’ve loaded this file into Kafka topic, I create a table in Hive.

Make sure that you have oracle-kafka.jar and kafka-clients*.jar in your hive.aux.jars.path:

and here:

after this you may run follow DDL in the hive:

hive> CREATE EXTERNAL TABLE json_web_clicks_kafka
row format serde 'oracle.hadoop.kafka.hive.KafkaSerDe'
stored by 'oracle.hadoop.kafka.hive.KafkaStorageHandler'
tblproperties(
'oracle.kafka.table.key.type'='long',
'oracle.kafka.table.value.type'='string',
'oracle.kafka.bootstrap.servers'='bds2:9092,bds3:9092,bds4:9092,bds5:9092,bds6:9092',
'oracle.kafka.table.topics'='json_clickstream'
);
hive> describe json_web_clicks_kafka;
hive> select * from json_web_clicks_kafka limit 1;

and as soon as hive table been created I create ORACLE_HIVE table in Oracle:

SQL> CREATE TABLE json_web_clicks_kafka (
topic varchar2(50),
partitionid integer,
VALUE  varchar2(4000),
offset integer,
timestamp timestamp, 
timestamptype integer
)
ORGANIZATION EXTERNAL
(TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR
   ACCESS PARAMETERS
      (
       com.oracle.bigdata.cluster=CLUSTER
       com.oracle.bigdata.tablename=default.json_web_clicks_kafka
      )
) 
PARALLEL 
REJECT LIMIT UNLIMITED;

here you also have to keep in mind that you need to add oracle -kafka.jar and  kafka -clients*.jar in your bigdata.properties file on the database and on the Hadoop side. I have dedicated the blog about how to do this here.

Now we are ready to query:

SQL> SELECT * FROM json_web_clicks_kafka
WHERE 
ROWNUM<3;

json_clickstream	209	{ click_date: "38041", click_time: "43213"..."}	0	26-JUL-17 05.55.51.762000 PM	1
json_clickstream	209	{ click_date: "38041", click_time: "74669"... }	1	26-JUL-17 05.55.51.762000 PM	1

Oracle 12c provides powerful capabilities for working with JSON, such as dot API. It allows us to easily query the JSON data as a structure: 

SELECT t.value.click_date,
       t.value.click_time
  FROM json_web_clicks_kafka t
 WHERE ROWNUM < 3;

38041	40629
38041	48699

Working with AVRO messages.

In many cases, customers are using AVRO as flexible self-described format and for exchanging messages through the Kafka. For sure we do support it and doing this in very easy and flexible way.

I do have a topic, which contains AVRO messages and I define Hive table over it:

CREATE EXTERNAL TABLE web_sales_kafka
row format serde 'oracle.hadoop.kafka.hive.KafkaSerDe'
stored by 'oracle.hadoop.kafka.hive.KafkaStorageHandler'
tblproperties(
'oracle.kafka.table.key.type'='long',
'oracle.kafka.table.value.type'='avro',
'oracle.kafka.table.value.schema'='{"type":"record","name":"avro_table","namespace":"default","fields":
[{"name":"ws_sold_date_sk","type":["null","long"],"default":null},
{"name":"ws_sold_time_sk","type":["null","long"],"default":null},
{"name":"ws_ship_date_sk","type":["null","long"],"default":null},
{"name":"ws_item_sk","type":["null","long"],"default":null},
{"name":"ws_bill_customer_sk","type":["null","long"],"default":null},
{"name":"ws_bill_cdemo_sk","type":["null","long"],"default":null},
{"name":"ws_bill_hdemo_sk","type":["null","long"],"default":null},
{"name":"ws_bill_addr_sk","type":["null","long"],"default":null},
{"name":"ws_ship_customer_sk","type":["null","long"],"default":null}
]}',
'oracle.kafka.bootstrap.servers'='bds2:9092',
'oracle.kafka.table.topics'='web_sales_avro'
);
describe web_sales_kafka;
select * from web_sales_kafka limit 1;

Here I define ‘oracle.kafka.table.value.type’=’avro’ and also I have to specify ‘oracle.kafka.table.value.schema’. After this we have structure.

In a similar way I define a table in Oracle RDBMS:

SQL> CREATE TABLE WEB_SALES_KAFKA_AVRO
   (  "WS_SOLD_DATE_SK" NUMBER, 
  "WS_SOLD_TIME_SK" NUMBER, 
  "WS_SHIP_DATE_SK" NUMBER, 
  "WS_ITEM_SK" NUMBER, 
  "WS_BILL_CUSTOMER_SK" NUMBER, 
  "WS_BILL_CDEMO_SK" NUMBER, 
  "WS_BILL_HDEMO_SK" NUMBER, 
  "WS_BILL_ADDR_SK" NUMBER, 
  "WS_SHIP_CUSTOMER_SK" NUMBER
  topic varchar2(50),
  partitionid integer,
  KEY NUMBER,
  offset integer,
  timestamp timestamp, 
  timestamptype INTEGER
   ) 
   ORGANIZATION EXTERNAL 
    ( TYPE ORACLE_HIVE
      DEFAULT DIRECTORY "DEFAULT_DIR"
      ACCESS PARAMETERS
      ( com.oracle.bigdata.tablename: web_sales_kafka
          )     
    )
   REJECT LIMIT UNLIMITED ;

And we good to query the data!

Performance considerations.

1) Number of Partitions.

This is the most important thing to keep in mind there is a nice article about how to choose a right number of partitions. For Big Data SQL purposes I’d recommend using a number of partitions a bit more than you have CPU cores on your Big Data SQL cluster.

2) Query fewer columns

Use column pruning feature. In other words list only necessary columns in your SELECT and WHERE statements. Here is the example.

I’ve created void PL/SQL function, which does nothing. But PL/SQL couldn’t be offloaded to the cell side and we will move all the data towards the database side:

SQL> create or replace function fnull(input number) return number is
Result number;
begin
Result:=input;
return(Result);
end fnull;

after this I ran the query, which requires one column and checked how much data have been returned to the DB side:

SQL> SELECT MIN(fnull(WS_SOLD_DATE_SK))
     FROM WEB_SALES_KAFKA_AVRO;

“cell interconnect bytes returned by XT smart scan” 5741.81MB

after this I repeat the same test case with 10 columns:

SQL> SELECT MIN(fnull(WS_SOLD_DATE_SK)),
       MIN(fnull(WS_SOLD_TIME_SK)),
       MIN(fnull(WS_SHIP_DATE_SK)),
       MIN(fnull(WS_ITEM_SK)),
       MIN(fnull(WS_BILL_CUSTOMER_SK)),
       MIN(fnull(WS_BILL_CDEMO_SK)),
       MIN(fnull(WS_BILL_HDEMO_SK)),
       MIN(fnull(WS_BILL_ADDR_SK)),
       MIN(fnull(WS_SHIP_CUSTOMER_SK)),
       MIN(fnull(WS_SHIP_CDEMO_SK))
  FROM WEB_SALES_KAFKA_AVRO;

“cell interconnect bytes returned by XT smart scan” 32193.98 MB

so, hopefully, this test case clearly shows that you have to use only useful columns

3) Indexes

There is no Indexes rather than Offset columns. The fact that you have key column doesn’t have to mislead you – it’s not indexed. The only offset allows you have a quick random access

4) Warm up your data

If you want to read data faster many times, you have to warm it up, by running “select *” type of the queries.

Kafka relies on Linux filesystem cache, so for reading the same dataset faster many times, you have to read it the first time.

Here is the example

- I clean up the Linux filesystem cache

dcli -C "sync; echo 3 > /proc/sys/vm/drop_caches"

- I tun the first query:

SELECT COUNT(1) FROM WEB_RETURNs_JSON_KAFKA t

it took 278 seconds.

- Second and third time took 92 seconds only.

5) Use bigger Replication Factor

Use bigger replication factor. Here is the example. I do have two tables one is created over the Kafka topic with Replication Factor  = 1, second is created over Kafka topic with ith Replication Factor  = 3.

SELECT COUNT(1) FROM JSON_KAFKA_RF1 t

this query took 278 seconds for the first run and 92 seconds for the next runs

SELECT COUNT(1) FROM JSON_KAFKA_RF3 t

This query took 279 seconds for the first run, but 34 seconds for the next runs.

6) Compression considerations

Kafka supports different type of compressions. If you store the data in JSON or XML format compression rate could be significant. Here is the examples of the numbers, that could be:

Data format and compression type Size of the data, GB
JSON on HDFS, uncompressed 273.1
JSON in Kafka, uncompressed 286.191
JSON in Kafka, Snappy 180.706
JSON in Kafka, GZIP 52.2649
AVRO in Kafka, uncompressed 252.975
AVRO in Kafka, Snappy 158.117
AVRO in Kafka, GZIP 54.49

This feature may save some space on the disks, but taking into account, that Kafka primarily used for the temporal store (like one week or one month), I’m not sure that it makes any sense. Also, you will pay some performance penalty, querying this data (and burn more CPU). 

I’ve run a query like:

SQL> select count(1) from ...

and had followed results:

Type of compression Elapsed time, sec
uncompressed 76
snappy 80
gzip 92

so, uncompressed is the leader. Gzip and Snappy slower (not significantly, but slow). taking into account this as well as fact, that Kafka is a temporal store, I wouldn’t recommend using compression without any exeptional need. 

7) Use parallelize your processing.

If for some reasons you are using a small number of partitions, you could use Hive metadata parameter “oracle.kafka.partition.chunk.size” for increase parallelism. This parameter defines a size of the input Split. So, if you set up this parameter equal 1MB and your topic has 4MB total, you will proceed it with 4 parallel threads.

Here is the test case:

- Drop Kafka topic

$   kafka-topics --delete --zookeeper cfclbv3870:2181,cfclbv3871:2181,cfclbv3872:2181 --topic store_sales

- Create again with only one partition

$   kafka-topics --create --zookeeper cfclbv3870:2181,cfclbv3871:2181,cfclbv3872:2181 --replication-factor 3 --partitions 1 --topic store_sales

- Check it

$   kafka-topics --describe --zookeeper cfclbv3870:2181,cfclbv3871:2181,cfclbv3872:2181 --topic store_sales
...
Topic:store_sales       PartitionCount:1        ReplicationFactor:3     Configs:
      Topic: store_sales      Partition: 0    Leader: 79      Replicas: 79,76,77      Isr: 79,76,77
...

- Check the size of input file:

$   du -h store_sales.dat
19G     store_sales.dat

- Load data to the Kafka topic

$   cat store_sales.dat|kafka-console-producer --broker-list cfclbv3870.us2.oraclecloud.com:9092,cfclbv3871.us2.oraclecloud.com:9092,cfclbv3872.us2.oraclecloud.com:9092,cfclbv3873.us2.oraclecloud.com:9092,cfclbv3874.us2.oraclecloud.com:9092 --topic store_sales  --request-timeout-ms 30000  --batch-size 1000000

- Create Hive External table

hive> CREATE EXTERNAL TABLE store_sales_kafka
row format serde 'oracle.hadoop.kafka.hive.KafkaSerDe'
stored by 'oracle.hadoop.kafka.hive.KafkaStorageHandler'
tblproperties(
'oracle.kafka.table.key.type'='long',
'oracle.kafka.table.value.type'='string',
'oracle.kafka.bootstrap.servers'='cfclbv3870:9092,cfclbv3871:9092,cfclbv3872:9092,cfclbv3873:9092,cfclbv3874:9092',
'oracle.kafka.table.topics'='store_sales'
);

- Create Oracle external table

SQL> CREATE TABLE STORE_SALES_KAFKA
   (	TOPIC VARCHAR2(50), 
      PARTITIONID NUMBER, 
      VALUE VARCHAR2(4000), 
      OFFSET NUMBER, 
      TIMESTAMP TIMESTAMP, 
      TIMESTAMPTYPE NUMBER
   ) 
   ORGANIZATION EXTERNAL 
    ( TYPE ORACLE_HIVE
      DEFAULT DIRECTORY DEFAULT_DIR
      ACCESS PARAMETERS
      ( com.oracle.bigdata.tablename=default.store_sales_kafka
    )     
    )
   REJECT LIMIT UNLIMITED 
  PARALLEL ;

- Run test query

SQL> SELECT COUNT(1) FROM store_sales_kafka;

it took 142 seconds

- Re-create Hive external table with ‘oracle.kafka.partition.chunk.size’ parameter equal 1MB

hive> CREATE EXTERNAL TABLE store_sales_kafka
row format serde 'oracle.hadoop.kafka.hive.KafkaSerDe'
stored by 'oracle.hadoop.kafka.hive.KafkaStorageHandler'
tblproperties(
'oracle.kafka.chop.partition'='true',
'oracle.kafka.partition.chunk.size'='1048576',
'oracle.kafka.table.key.type'='long',
'oracle.kafka.table.value.type'='string',
'oracle.kafka.bootstrap.servers'='cfclbv3870:9092,cfclbv3871:9092,cfclbv3872:9092,cfclbv3873:9092,cfclbv3874:9092',
'oracle.kafka.table.topics'='store_sales'
);

- Run query again:

SQL> SELECT COUNT(1) FROM store_sales_kafka;

Now it took only 7 seconds

One MB split is quite low, and for big topics we recommend to use 256MB.

8) Querying small topics.

Sometimes it happens that you need to query really small topics (few hundreds of messages, for example), but very frequently. At this case, it makes sense to create a topic with fewer paritions.

Here is the test case example:

- Create topic with 1000 partitions

$   kafka-topics --create --zookeeper cfclbv3870:2181,cfclbv3871:2181,cfclbv3872:2181 --replication-factor 3 --partitions 1000 --topic small_topic

- Load only one message there

$   echo "test"|kafka-console-producer --broker-list cfclbv3870.us2.oraclecloud.com:9092,cfclbv3871.us2.oraclecloud.com:9092,cfclbv3872.us2.oraclecloud.com:9092,cfclbv3873.us2.oraclecloud.com:9092,cfclbv3874.us2.oraclecloud.com:9092 --topic small_topic

- Create hive external table

hive> CREATE EXTERNAL TABLE small_topic_kafka
row format serde 'oracle.hadoop.kafka.hive.KafkaSerDe'
stored by 'oracle.hadoop.kafka.hive.KafkaStorageHandler'
tblproperties(
'oracle.kafka.table.key.type'='long',
'oracle.kafka.table.value.type'='string',
'oracle.kafka.bootstrap.servers'='cfclbv3870:9092,cfclbv3871:9092,cfclbv3872:9092,cfclbv3873:9092,cfclbv3874:9092',
'oracle.kafka.table.topics'='small_topic'
);

- Create Oracle external table

SQL> CREATE TABLE small_topic_kafka (
topic varchar2(50),
partitionid integer,
VALUE varchar2(4000),
offset integer,
timestamp timestamp,
timestamptype integer
)
ORGANIZATION EXTERNAL
(TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR
ACCESS PARAMETERS
(
com.oracle.bigdata.tablename=default.small_topic_kafka
)
)
PARALLEL
REJECT LIMIT UNLIMITED;

- Query all rows from it

SQL> SELECT * FROM small_topic_kafka

it took 6 seconds

- Create topic with only one partition and put only one message there and run same SQL query over it

$   kafka-topics --create --zookeeper cfclbv3870:2181,cfclbv3871:2181,cfclbv3872:2181 --replication-factor 3 --partitions 1 --topic small_topic
  
$   echo "test"|kafka-console-producer --broker-list cfclbv3870.us2.oraclecloud.com:9092,cfclbv3871.us2.oraclecloud.com:9092,cfclbv3872.us2.oraclecloud.com:9092,cfclbv3873.us2.oraclecloud.com:9092,cfclbv3874.us2.oraclecloud.com:9092 --topic small_topic
SQL> SELECT * FROM small_topic_kafka

now it takes only 0.5 second

9) Type of data in Kafka messages.

You have few options for storing data in Kafka messages and for sure you want to do pushdown processing. Big Data SQL supports pushdown operations only for JSONs. This means that everything that you could expose thought the JSON will be pushed down to the cell side and will be prosessed there.

Example

- The query which could be pushed down to the cell side (JSON):

SQL> SELECT COUNT(1) FROM WEB_RETURN_JSON_KAFKA t
WHERE
t.VALUE.after.WR_ORDER_NUMBER=233183247;

- The query which could not be pushed down to the cell side (XML):

SQL> SELECT COUNT(1)
  FROM WEB_RETURNS_XML_KAFKA t
 WHERE 
XMLTYPE(t.value).EXTRACT('/operation/col[@name="WR_ORDER_NUMBER"]/after/text()')
 .getNumberVal() = 233183247;

If amounts of data is not significant, you could use Big Data SQL for processing. If we are talking about big data volumes, you could process it once and convert into different file formats on HDFS, with Hive query:

hive> select xpath_int(value,'/operation/col[@name="WR_ORDER_NUMBER"]/after/text()') from WEB_RETURNS_XML_KAFKA limit 1 ;

10) JSON vs AVRO format in the Kafka topics

In continuing to the previous point, you may be wondering which semi-structured format use? The answer is easy – use what your data source produce there is no significant performance difference between Avro and JSON. For example, a query like:

SQL> SELECT COUNT(1) FROM WEB_RETURNS_avro_kafka t
WHERE
t.WR_ORDER_NUMBER=233183247;

Will be done in 112 seconds in case of JSON and in 105 seconds in case of Avro.

and JSON topic will take 286.33 GB and Avro will take 202.568 GB. There is some difference, but not worth for converting the original format.

How to bring data from OLTP databases in Kafka? Use Golden Gate!

Oracle Golden Gate is the well-known product for capturing commit logs on the database side and bring the changes into a target system. The good news that Kafka may play a role in the target system. I’d like to skip the detailed explanation of this feature, because it’s already explained in very deep details here.

Let’s block ads! (Why?)

Oracle Blogs | Oracle The Data Warehouse Insider Blog

Released: SQL Server 2017+ and Replication Management Packs (7.0.0.0)

We are happy to announce the final release of SQL Server 2017+ and Replication Management Packs! These MPs can be used to monitor SQL Server 2017 on both Windows and Linux and SQL Server 2017 Replication on Windows. Below, let’s look at some of the new features we added to these MPs. All the details regarding the new functionality can be found in the Operations Guide that can be downloaded along with the Management Pack.

Cross-platform Monitoring

You can now monitor SQL Server 2017 on Windows and on Linux!

Version Agnostic SQL Server MPs

We understand that with many SQL Server versions in market and with new server releases becoming more frequent, it is becoming harder to manage a separate MP for each server version. We are moving to version agnostic MPs to address this issue. This will be valid going forward. The new MP is named SQL Server 2017+. The ‘+’ in the name indicates that it will be used to monitor SQL Server 2017 and the releases that come after that. Current in-market MPs (2008 through 2016) will not be changed and the 2017+ MP cannot be used to monitor older releases. We are making this change for SQL Server and Replication MPs.

Agentless Monitoring

SQL Server 2017+ Management Pack introduces Agentless Monitoring mode support. This monitoring mode is designed to support SQL Server on Linux but it also works for Windows deployments. We’ve received multiple requests from our customers to support agentless monitoring for SQL Server that can be useful for production OLTP workloads or in case the organization’s policy denies deployment of SCOM agent to the monitored host or making any changes to it.

With the Agentless Monitoring mode there is no need to deploy SCOM Agent on the SQL box. Instead, the monitoring workloads will be transferred to management servers included in the SQL Server Monitoring Pool. This allows to remove SCOM Agent and data processing overhead from the SQL box and move it to the SQL Server Monitoring Pool. Also with the agentless monitoring mode, you can use SQL credentials for authentication that simplifies security configuration.

Engineering Improvements

  • Usage of scripts is discontinued in favor of .Net Framework modules, which enables more efficient resource usage.
  • For getting information on health and performance, SQL Server Dynamic Management Views and Functions are now used instead of WMI calls. This provides better efficiency and connectivity.

Downloads available:

Microsoft System Center Management Pack for SQL Server 2017+

Microsoft System Center Management Pack for SQL Server 2017+ Replication

Microsoft System Center Management Pack for SQL Server Dashboards

We are looking forward to your feedback.

Let’s block ads! (Why?)

SQL Server Release Services

Digitalist Magazine

Rapid change will continue due to ongoing digital transformation. What are the possible outcomes? Which future will happen? How are you prepared?

Let’s block ads! (Why?)

Digitalist Magazine

Top Ten Digitalist Magazine Posts Of The Week [November 13, 2017]

In May 2017, a computational social scientist from The Psychometrics Centre at the University of Cambridge stood before an audience at the Linux Foundation’s Apache Big Data conference and revealed how close we’ve come to the ultimate goal of marketing: an easily scalable, highly accurate way to predict customer preferences using minimal data.

When she was still a PhD candidate, Sandra Matz created a Facebook ad campaign targeting people based on nothing more than how extroverted their Facebook Likes indicated they were. People with Likes associated with extroverts saw ads for a party game played in a group. People with more introverted Likes saw ads for a quiet game meant to be played solo.

The campaign required only simple algorithms and no advanced analytics. Yet over seven days of testing, the targeted ads generated up to 15 times higher click-through and conversion rates—and significantly more purchases and revenue for the game company.

SAP Q317 DigitalDoubles Feature3 Image2 Top Ten Digitalist Magazine Posts Of The Week [November 13, 2017]“We developed this approach to show that you can achieve highly accurate behavioral and psychological targeting with a minimal amount of data and relatively simple machine learning tools,” says Matz, who is now an assistant professor of management at Columbia University’s business school.

As effective as this experiment was, Matz suggests that it’s still rudimentary compared to what could be done with more and richer data from more sources. And it’s downright primitive given the possibilities of applying more sophisticated Big Data analytics.

These possibilities have created a watershed moment for marketing and its role in the business.

Spiraling Down the Marketing Funnel

Tension has always simmered over marketing’s contribution to business success. The business knows it can’t sell products or services if it doesn’t make customers aware of them, but the impact of marketing strategy on sales and revenue is hard to quantify and reliably replicate—which, in the age of the data-driven enterprise, often leaves some business leaders not just undervaluing marketing but actively mistrusting it. No wonder human resources consultancy Russell Reynolds reports that the 2016 turnover rate among CMOs was the highest it has seen since it began tracking the statistic in 2012.

Most companies still determine customers’ readiness to buy by using a primitive model known as the marketing funnel, which sorts customers into increasingly smaller groups as they progress from first becoming aware of a company to buying, using, and finally advocating for the company’s products. Different versions have different definitions and numbers of stages, and some approaches see the model as a circle, but they all have one thing in common: their ability to sort customers into various stages is limited by the amount of knowledge the company has about each customer.

As a result, the marketing funnel ends up leaking. Some customers back away because they feel harassed by campaigns that don’t apply to their needs, while some of those who are interested fall through the cracks from a lack of attention. Many data-hungry business leaders think of the marketing funnel as no more than a variation of “throw something against the wall and see if it sticks,” and with the proliferation of digital channels and diffusion of customer attention, they have less patience than ever with that approach.

The silver lining is that a more precise, quantifiable way to build customer relationships is emerging. Done properly, it promises to defuse the tension between marketing and the rest of the business, too.

SAP Q317 DigitalDoubles Feature3 Image3 1024x572 Top Ten Digitalist Magazine Posts Of The Week [November 13, 2017]

The Defining Moment

The Cambridge University experiment is one more step toward the long-held marketing dream of the “segment of one.” This concept of marketing messages that are highly granular, even individually tailored, has been around since the late 1980s. Over the last 15 to 20 years, as customer behavior has become digitalized as never before, marketers have been optimistic that they could capture this data and use it to tailor their messaging with laser-like precision.

Yet what’s achievable in theory has been impossible in practice. We’re still struggling to find the right tools to move beyond the basics of demographic targeting. The rise of the internet, smartphones, and social media has generated more types of information about customer behavior in larger amounts than ever before. But using digitally expressed sentiment about everything from toys to turbines as the basis for accurately disseminating highly individualized marketing messages is still time consuming and cost prohibitive.

However, experiments like Matz’s are bringing us closer to creating highly personalized customer experiences—perhaps not always at the individual level but certainly at a level of granularity that will let us unequivocally determine how to best target and measure marketing programs.

Liking Lady Gaga

Between 2007 and 2012, Psychometrics Centre researchers gathered seven million responses to a simple questionnaire for Facebook users. The carefully designed questions measured respondents’ levels of extroversion, agreeableness, openness, conscientiousness, and neuroticism, a constellation of basic personality traits known as the Big Five.

With the respondents’ permission, the researchers used simple machine learning tools to correlate each person’s responses with the official Facebook Pages that the person had liked, such as Pages for books, movies, bands, hobbies, organizations, and foods. They soon saw that certain personality traits and certain Likes went hand in hand.

For example, most people who liked Lady Gaga’s Page tested as extroverts, which made liking the Lady Gaga Page a relevant data point indicating that someone was probably an extrovert. By 2016, Matz was able to create a lively Facebook ad to be shown only to people who had liked a significant number of official Pages that seemed to be linked to extroversion. A more serene ad was shown only to those whose Likes suggested that they were introverts.

SAP Q317 DigitalDoubles Feature3 Image4 Top Ten Digitalist Magazine Posts Of The Week [November 13, 2017]Despite the large size of the Psychometric Centre’s data set, what’s most remarkable about its work is how few data points within that data set were necessary to build a reliable profile that could model useful predictions. Matz told EnterpriseTech that the algorithm the Centre developed needs, on average, just 65 liked Pages to understand someone’s Big Five personality traits better than their friends do, 120 to understand them better than their family members, and 250 to understand them better than a partner or spouse. This may be the first sign that the era of true behavioral marketing is upon us.

Of course, most marketers want to know far more about customers than how outgoing or reserved they are. Scraping Facebook Likes isn’t enough to give them the holistic customer understanding they crave—not when they have an entire universe of other data to consider. The race is on to identify from the vast spectrum of available customer data not only which specific online behaviors have a predictive element such as extroversion or introversion but also which ones will drive the most potent response to specific product or service messaging.

Complicated? Yes—but we are within reach of the algorithms we need to connect the dots for greater customer insight. By reaching out over new channels with more accurate behavior-based messaging, companies could transform the entire customer journey.

A Customized Journey for Each Customer

Attribution, the ability to know the source of a sales lead, is key to behavioral targeting. The more details a business knows about what its customers have already done, the more accurately it can predict what they will do next.

In the past, developing a customer profile relied on last-touch attribution analysis, that is, evaluating the impact of the last interaction a prospective customer had with a brand before becoming a lead. The problem was that companies could rarely be certain what that last touch was, given how much activity still takes place offline and isn’t captured or quantified.

Companies also couldn’t be certain how, or even if, a last touch—be it downloading a white paper, visiting a store, or getting a word-of-mouth recommendation—accelerated the customer through the marketing funnel. They could only predict revenue by looking at how many people were deemed to be at a specific stage and extrapolating from past data what percentage of them were likely to move ahead.

SAP Q317 DigitalDoubles Feature3 Image5 Top Ten Digitalist Magazine Posts Of The Week [November 13, 2017]Today, we’re capturing so much more information about people’s activities that we have a far more accurate idea of both what the last touch was and how influential it was. Behavioral targeting makes any content a customer interacts with valuable in analyzing the customer’s journey. A company can use hard data about those interactions to see where each individual prospect is in the customer journey and predict how likely each one is to continue moving forward.

The company can then generate a tailored offer or other event to nudge individuals along based on what has been successful with other customers who buy the same things and behave in the same ways. For example, a large grocer may send out two million individualized offers each week based on loyalty card activity. This may not strictly create a segment of one, but it creates many small segments of customers with similar behaviors based on what the grocer knows to be effective.

As Cambridge University’s experiment in creating an algorithm to identify and target introverts and extroverts proves, more precise messaging is more effective. By using more complex machine learning algorithms to further filter and refine successful messages to target smaller groups, companies could boost their conversion rates to as high as 50%—an exponential increase beyond today’s average rates.

By using machine learning to speed up the testing of different campaigns and to continuously compare results, companies could rapidly create a dataset about every potential customer’s responses and then benchmark it against others’ responses. This would let them determine individual prospects’ likely responses based on concrete actions rather than assumptions.

For super-luxury brands with a limited number of customers and the ability to capture a vast amount of information about each one, this could lead to true segment-of-one marketing. For other brands, the challenge is not just to figure out who the customer is and what messages to send but also how to scale that personalization to segments of tens of thousands (or hundreds of thousands) of customers at a time. To do that both effectively and quickly, companies will need to leverage machine learning, the Internet of Things, and other advanced technologies that enable accurate predictive models. Companies can then benchmark their projected hit rates against their actual results and refine their algorithms for even greater agility and responsiveness.

The Next Steps of Predictive Marketing

Effective behavioral targeting requires companies to identify all the relevant data points, including external data points that indicate which information is valuable. This calls for data scientists who can spot and remove the irrelevant data points that are at the far ends of the curve and distill what remains into meaningful algorithms. It also requires machine learning tools capable of high-volume, high-speed listening, assessing, learning, and making recommendations to improve the algorithm over time.

Once you’ve created a baseline of primary criteria, you can determine the important criteria by which to segment your customer base. To use an oversimplified example, imagine that you own a coffee shop and you want to increase sales of high-margin bakery items. You need to look not at the customers who always get a muffin with their coffee or at those who never do but at those who buy a muffin sometimes, so that you can start to identify the triggers that make them choose to indulge.

To scale this process, look at both user-based and item-based affinities. User-based affinities link customers who have similar interests and shopping patterns. Item-based affinities link customers based on what they buy, individually or in groups of items. Using machine learning to pair and cross-reference these two factors will enable you to create messages that are personalized enough to seem individualized, even though they’re actually targeting small, multi-person segments.

SAP Q317 DigitalDoubles Feature3 Image6 Top Ten Digitalist Magazine Posts Of The Week [November 13, 2017]Retailers of all types collect data about individuals, down to location, date, time, and SKU of the sale. They may experiment with behavioral targeting by making in-the-moment offers based on what they already know about their customers. For example, they may use a mobile app with geofencing to be alerted when a customer using the app is in the store. The alert triggers back-end systems to look up the customer’s purchase history, generate a relevant offer, and deliver that offer to the customer’s smartphone while the customer is still in the store.

The Line Between Marketing and Manipulation

Just the idea of receiving marketing messages influenced by their behavior will disturb some customers. When marketing is designed, as behavioral targeting is, to maximize engagement, the value of the content depends less on whether it’s useful to the audience or even true and more on whether it gets the target audience to engage and reveal another piece of the behavioral puzzle. As a result, companies considering behavioral marketing must consider a question as old as marketing itself: where is the line between advertising and propaganda?

Creating personal profiles of customers based on their actions and personalities will become inexpensive and easy, for better or worse. Better will lead to more relevant and compelling offers based on predictive models of what customers would like to buy next. Worse will create (or at least look like) scalable, granular manipulation.

If companies hope to apply this level of targeted marketing without coming across as intrusive or invasive, they will need to be completely transparent about what they’re doing and how—and with whom they’re sharing the information. Most shoppers say they’re willing to give up data about themselves if it leads to a better shopping experience and more relevant recommendations.

Numerous studies show that customers are comfortable sharing their buying patterns and preferences as long as it doesn’t compromise their personally identifiable information. Nonetheless, they may decide otherwise if they believe that by welcoming you into their lives, they’re throwing open the doors to strangers as well.

SAP Q317 DigitalDoubles Feature3 Image7 1024x572 Top Ten Digitalist Magazine Posts Of The Week [November 13, 2017]

As data mining for behavioral targeting becomes more common, companies will have to offer customers the opportunity to opt in and out at varying levels of detail. They will also need to identify and flag the significant minority of customers who prefer not to be profiled in such depth (or at all). Machine learning will be invaluable in responding to complaints on social media, tracking the relevant details of offers that were ignored or got negative reactions, and otherwise ensuring that companies don’t misuse customer data or misunderstand consumer wants and needs.

“The entire paradigm of targeting and campaign implies a vendor doing something to customers,” says Mark Bonchek, founder and “chief epiphany officer” at Shift Thinking, a Boston-based consulting firm that helps companies pursue digital transformation. “It implies getting people to do what you want them to do rather than helping them do what they want to do,” he says. “Be clear on the mental model behind your behavioral targeting. Is it more like a friend figuring out the right gift for a friend or a salesperson trying to close a deal with a prospect? People don’t want to be targets.”

Instead, Bonchek suggests, think of behavioral targeting as a way to build a reciprocal relationship that lets you enhance the customer experience at multiple touch points, not all of them actual transactions. Utility companies send customers information about their own and their neighbors’ energy use so they can benchmark themselves. The utilities often follow up with suggestions about how to save both power and money. Meanwhile, a credit card issuer could help customers understand their purchasing patterns and discover new stores or service providers.

“Loyalty is an emotion first and behavior second,” Bonchek says. “It’s the difference between pushing customers through a funnel and helping them achieve a shared purpose.”

The Art of Scientific Marketing

In mid-20th century New York City, a small local chain of markets developed a national reputation for customer service. It let favored customers call in orders and pay for them at pickup. Managers kept lists—handwritten lists, no less—of their best customers’ preferred products and called those customers with special offers. People were happy to pay slightly higher prices overall in exchange for exclusive bargains and highly customized service.

Although it leverages new technologies like machine learning and Big Data, behavioral targeting will in many ways bring us full circle to that hands-on era in which companies created relevant offers that made customers feel valued and understood. Matz believes it would be a competitive advantage for companies to let customers interact with their profiles and even correct them to ensure that they only receive offers that meet their needs and preferences.

As more situational data pours in from smartphones and wearables to be analyzed by AI, she adds, behavioral targeting could become something more immersive than mere marketing. “If you know from that data that someone is not just an extrovert with specific preferences but that they’re currently in a good mood, you can start fine-tuning messages for that particular point in time,” she says. “We’ll move beyond static profiles to interactions based on characteristics that fluctuate.”

With enough data to work with, she suggests, behavioral targeting could become less about making offers and more about informing customers about their options at any given moment, in real time. D!


About the Authors

Denise Champion is Vice President of Strategy, Research, and Insights for Global Marketing at SAP.

Jeff Harvey is Global COO, SAP Analytics & Insight at SAP.

Lori Mitchell-Keller is Global General Manager, Consumer Industries at SAP.

Jeff Woods is Global COO, SAP Leonardo | Data and Analytics.

Fawn Fitter is a freelance writer specializing in business and technology.


Read more thought provoking articles in the latest issue of the Digitalist Magazine, Executive Quarterly.

Comments

Let’s block ads! (Why?)

Digitalist Magazine

Calculation of $P( X>- {\cal E}[2^{-x}] | X

A random variable $ X$ has a cumulative distribution function $ F$ defined by a formula:

6vc0r Calculation of $P( X>  {\cal E}[2^{ x}] | X

I do not know how can I calculate

$ $ P(X>{\cal E}[2^{-X}]|X<{\cal E}[X^3])$ $ ,

where $ {\cal E}[\cdot]$ is the expectation operator.

1 Answer

Let’s block ads! (Why?)

Recent Questions – Mathematica Stack Exchange

Jen Cohen Crompton – Digitalist Magazine

It’s that time of the year again and we are all checking our lists and checking them twice…to make sure we didn’t forget to purchase any last minute items for our loved ones who may have been naughty or nice (but either way, we still have to buy them something, right?)

Well if you’ve hit mid-December and still have a few people to check off this year and you’re stumped trying to figure out what Santa should bring, here are a few ideas for any tech-lover from ages five to 95.

For the music lovers

This year, just about Bluetooth-enabled will be a hit. With the adoption of smartphones hitting an all-time high, most of us are completely attached to these gadgets and products that help us get the most out of them seem ingenious.

The Ultimate Ears (UE) UE Boom portable speakers are probably the best Bluetooth product boom Jen Cohen Crompton – Digitalist Magazinespeakers on the market. Although rivals Bose and Beats have Bluetooth speakers that are better known, the UE brand offers more capabilities and a better sound. The UE Bluetooth speakers take moments to set up and through the app, you can hook up more than one speaker and create a stereo sound (something the other speakers don’t offer). Oh and did we mention that the speaker is waterproof? Think outdoor parties and being able to leave your smartphone indoors so it doesn’t get damaged. The UE Boom retails at $ 199 and comes in a variety of colors and even offers some with designs.

For teenagers or someone looking for a more visual appeal (and a product around the same price point), the JBL Pulse Bluetooth speaker absolutely has that visual side. With LED lights that move to the music, this pill-shaped device makes more than just music – it makes a statement.

For the one who hates wires

For years, products have tried to remove the BlueBuds X Storm White Packaging 300x300 Jen Cohen Crompton – Digitalist Magazinewires connecting our ears to our music players, but without much success and with compromising quality. However, this year, a few wireless headphone companies have come through in a big way and one of the best is Jaybird with the Bluebuds X. These beautifully designed wireless earbuds alleviate the confusion as to where you store your super gigantic Phablet-like iPhone 6 Plus while trying to take a stroll on the treadmill – you can store anywhere within the detectable distance and still hear great sound.

These headphones are great for anyone who enjoys working out or an on-the-go lifestyle and doesn’t want wires getting in the way. Although the $ 170 price tag is a little higher than its competitors, the quality and fit of these earbuds also seem to surpass competitors, so they could be worth the investment.

For the phone dead felon

Hate hearing the excuse, “my phone was dead?” In a world full of not having a charger or not being 1350x1240 wysiwyg helium ip5s Front Back 3QTR3 100713 300x275 Jen Cohen Crompton – Digitalist Magazineable to find an outlet, but living and dying by our phones, a better way to charge our lifelines is a must.

There are plenty of phone charging cases, but one of the best is the Mophie. The Mophie is a battery pack on steroids and versions of the product can bring your phone from dead to 100 percent in minutes (and some even offer additional storage). The case has versions for all types of devices and all it takes is a quick flip of the switch to bring power back to your life. The products hover around the 80-dollar mark.

For the organized (or wannabe organized) colleague or friend

Know someone who despises clutter and would love to get their paperwork more organized? Thanks to our digital lives and the ability to transmit data paperlessly, now is as good of a time as any to convert paper files into digital ones and consider decluttering all aspects of lives.

A great product to make this happen is receipts 610px receipt 300x235 Jen Cohen Crompton – Digitalist MagazineNeatReceipts, which is a portable scanner and smart organization system. All you have to do is scan the documents using the device and then the digital images are stored within the software where they become filed and searchable. Another awesome option about this product is that it does come with an app and you can use the app to scan images and the data will still be parsed accordingly. The NeatReceipts retails at about $ 180.

For the kids

And of course, we cannot forget the kids when we talk holiday gifts and technology. Of course there are plenty of mobile devices and tablets that are lots of fun, but let’s bring it back to good old toys – that are now synced with tech.

Check out Zoomer Dino. Zoomer is a robotic dinosaur that operates by detecting hand dino deco1 300x210 Jen Cohen Crompton – Digitalist Magazinegestures, or by remote control. Charged using a USB port, Zoomer uses its head sensors to detect movement and will respond accordingly based on some pre-programmed commands. However, Zoomer is a dinosaur so he can be somewhat unpredictable and get angry and go a bit crazy by spinning around and turning his eyes red. Overall, Zoomer is a pretty good time and he retails for about $ 80-100.

For kids who want a furry friend and parents who don’t want to have to take care of one, The FurReal Friends Get Up and GoGo My Walkin’ Pup Pet is a lifelike dog that barks, wags its tail, char page gogo 298x300 Jen Cohen Crompton – Digitalist Magazineand walks – but does none of the “un-fun” stuff and can easily be turned off. Furreal friends respond to voice prompts and interact with each other and the compatible app when downloaded and activated. FurReal pets can be found around the $ 50 price point.

There are plenty of other amazing tech toys out this season, so use these as suggestions and shop around to find some deals. After all, Santa may still come down the chimney, but he’d probably much rather be delivering presents while riding one of these.

]]>

http://www.digitalistmag.com/industries/retail/2014/12/05/week-tech-black-friday-results-google-gets-friendly-lights-sony-struggles-01885247</link>
http://www.digitalistmag.com/industries/retail/2014/12/05/week-tech-black-friday-results-google-gets-friendly-lights-sony-struggles-01885247#comments Fri, 05 Dec 2014 18:00:21 +0000http://blogs.sap.com/innovation/?p=1885247This week in tech revealed the results from the holiday shopping kickoff weekend, Google Friday Flashback11 Jen Cohen Crompton – Digitalist Magazinemaking strides to become more kid-friendly and inspire coding, and Sony struggling with their recent hack that has gone from bad to worse.

Here’s what happened, this week in tech.

1. The Mad Rush to Shop…Online. After the busiest shopping weekend of the year, we have the results of this four-day sale phenomenon known as Black Friday weekend.

According to Fundivo.com, this year, Black Friday spending reached $ 50.9 billion, which was lower than last year, but Black Friday online shopping went from $ 1.198 billion in 2013 to $ 1.505 billion this year. Cyber Monday sales showed success as they were up from the $ 2.290 billion spent in 2013, to $ 2.680 billion this year. Overall, online sales were a huge success for most retailers, while in-store sales weren’t as profitable. One retailer that needed to pull in some sales and did exceptionally well was Walmart. Walmart.com experienced over 1.5 billion pages viewed by customers, and “70 percent of its traffic coming from mobile devices.”

According to IBM and its Benchmark tool that tracks sales in real time, in total online sales were up 17 percent over last year and mobile continued to be a sales driver accounting for 51.2 percent of ecommerce browsing and 28.9 percent of sales.

However, some companies, such as Best Buy, experienced a website crash. During Black Friday (and actually for a bit on Saturday morning), Best Buy’s website was unavailable. A spokesperson for Best Buy reported that “a concentrated spike in mobile traffic triggered issues that led us to shut down BestBuy.com in order to take proactive measures to restore full performance.” Fortunately though, Best Buy crashed due to high traffic and although they haven’t reported their numbers, it is predicted that the outage wasn’t that detrimental to their overall four-day sales total.

2. Google Gets Friendlier. Thanks to a focus on cybersecurity and keeping children out of harm’s way on the net, Google is now working on a kid-friendly version of its search engine. Since kids are big consumers of technology, being born after the advent of widespread technological adoption, Google searches from kids 12 and under are on the rise. This new focus for Google could include kid-friendly versions of YouTube and Chrome, with controls for parents to help manage what their kids may see and do. The vice president of engineering who is leading the way, Pavni Diwanji, stated that, “the big motivator inside the company is everyone is having kids, so there’s a push to change our products to be fun and safe for children.” There’s no official word on a launch date, but it’s rumored that this feature may make its debut sometime next year.

3. Light It Up. Speaking of kid-friendly, Google recently rolled out their Made with Code program to help girls become more interested in technology. Over 300,000 people – mostly young girls – helped design the lighting display for the 92nd Annual White House Christmas Tree Lighting Ceremony, which includes 56 official White House Christmas Trees. Girls ages four through 20 participated in writing the code to bring the Christmas Trees to life. One participant and Made with Code ambassador, Brittany Wegner, 20, will join ten other Made with Code girls to watch the lighting extravaganza. She explains that each code created will have a specific time for its debut, “down to the exact second.”

Made to Code helps introduce coding to girls at a young age by showing the different possibilities there are to using this incredible skill. In Wegner’s words, “Made with Code is more of an introductory learning platform to get girls interested in coding so it makes it super easy.” Other girls wanting to be part of lighting up the White House this Christmas is encourage to participate through Google’s Made with Program. The White House trees will continue to add more programs for future submissions for those that missed the lighting ceremony.

4. The Sony Struggle. Last week, Sony was hacked and the initial thought was that the hack was possibly a result of misuse internal sources. Unfortunately, the story is unraveling and Sony has a bigger issue to deal with this week.

This week, movies were leaked and employee information (think social security numbers, salaries, health information, etc.) was exposed to the public. The hack was so serious, that the FBI is now warning other companies to keep a watchful eye on their data. Memos were also leaked from employees who are less than satisfied with Sony’s performance and this sensitive data puts Sony at risk for losing money, respect, and possibly employees. The salary leak revealed uneven gender and race paychecks, and exposed employee information puts thousands of Sony employees at risk for identity theft. With the threat of taking a year to sort through and deal with the hack, the fingers are now pointed at North Korea who may be the source of the attack as a rebuttal against the soon-to-be release of the film “The Interview,” in which the characters plot to kill North Korean leader Kim Jong-Un. Yikes!

That’s all for this week in tech. Did we miss anything?

]]>http://www.digitalistmag.com/industries/retail/2014/12/05/week-tech-black-friday-results-google-gets-friendly-lights-sony-struggles-01885247/feed11885247 http://www.digitalistmag.com/industries/sports-and-entertainment/2014/12/01/last-week-tech-twitter-fail-sony-gets-hacked-netflix-investigates-best-buy-crashes-01864673</link>
http://www.digitalistmag.com/industries/sports-and-entertainment/2014/12/01/last-week-tech-twitter-fail-sony-gets-hacked-netflix-investigates-best-buy-crashes-01864673#comments Mon, 01 Dec 2014 15:39:01 +0000http://blogs.sap.com/innovation/?p=1864673Although last week was full of being thankful and hopefully some tasty turkey consumption, Friday Flashback11 Jen Cohen Crompton – Digitalist Magazinethe tech world didn’t take time off.

Last week in tech, Twitter’s CFO made a little mistake that ended up on his Twitter page while Sony Pictures had some internal employee problems with hacking. There was also a former Netflix employee who went a little too far and Best Buy realizing they were ill-prepared for the biggest shopping day of the year.

Here’s a recap of what happened last week in tech.

1. #Tweetfail. Twitter’s CFO, Anthony Noto, accidentally tweeted a private message for the whole world to see. The public tweet, intended to be a direct message, read: “I still think we should buy them. He is on your schedule for Dec 15 or 16 — we will need to sell him. i have a plan.” Although it is now deleted, someone with a quick eye took a screenshot of his tweet making him fall prey to the “whatever you put on the Internet stays on the Internet” philosophy. There is no confirmation on the identity of the company Noto tweeted about, but it sure put the CFO’s skills into question and wondering if he knows how to use his own service…or maybe he only knows how to use it as well as Anthony Weiner. Unfortunately, this failed tweet came at a very inconvenient time and didn’t help Twitter gain any headway after their shares plummeted by 17 percent. Hopefully whatever the planned acquisition was for Twitter, it will help turn things around and meanwhile, Noto will get schooled on how to use Twitter’s private messaging system.

2. Sony Pictures Gets Hacked. While most hacker stories this year have been from outside sources or other countries, the hack on Sony Pictures appears to be an inside job. Last week, a massive data breach shut down Sony Pictures. These, supposed internal, hackers are threatening to share the company’s top secrets if their demands are not met. Reddit user and ex-Sony employee uploaded the image claiming that it appeared on all company computers. Sony Pictures is claiming this to be a “system disruption” and the matter is being investigated.

3. Netflix Sues Former Employee – A Netflix former executive, Mike Kail, was discovered taking kickbacks from outside suppliers. His job at Netflix as an IT employee was to buy outside services and computer gear. By taking advantage of his authority and charging referral fees from suppliers, he was able to take a pretty penny for himself. Exactly how many pennies? From two firms that Netflix paid $ 3.7 million for their services, Kail’s kickback was $ 490,000 – that’s a lot of pennies. Netflix is suing the now Yahoo’s chief information officer for charging this extra fee, which would be anywhere from 10 to 15 percent of every invoice.

The investigation began when Kail left to join Yahoo. Some Netflix employees questioned a few of the details of the contracts with outside companies and when Netflix tried to review those contracts (that were of course, signed by Kail) they realized he had locked Netflix out of his accounts. Kail used his consulting company, Unix Mercenary, which is conveniently not listed on Kail’s resume and does not have any records of existing. Neither Netflix, Yahoo, nor Kail had any comment on the lawsuit.

4. Best Buy Crashes. On one of the busiest shopping days of the year, you won’t make it into the black if the website crashes. Unfortunately, this is what Best Buy experienced this past Black Friday. Best Buy has been doing very well with online sales, up 20 percent, compared to their in-store sales, which were down 2 percent last Black Friday. A spokesperson for Best Buy says that “a concentrated spike in mobile traffic triggered issues” led them to shut down BestBuy.com and they took proactive measures to restore full performance. Unfortunately for mobile shoppers, the app was also affected. But, there was sunshine at the end of the storm since the site and app were back up by 11:30 a.m. ET…but of course, there was still quite a good amount of time that Best Buy was out of commission and could have been bringing in those sales. Hopefully, Best Buy learn from this and can work out the bugs and not crash today (Cyber Monday) since 44 percent of America plans to buy online this popular shopping weekend!

That’s it for last week in tech. Did we miss anything? Let us know.

]]>http://www.digitalistmag.com/industries/sports-and-entertainment/2014/12/01/last-week-tech-twitter-fail-sony-gets-hacked-netflix-investigates-best-buy-crashes-01864673/feed11864673 http://www.digitalistmag.com/technologies/mobile-applications/2014/11/26/5-essential-black-friday-apps-01833697</link>
http://www.digitalistmag.com/technologies/mobile-applications/2014/11/26/5-essential-black-friday-apps-01833697#comments Wed, 26 Nov 2014 13:00:00 +0000http://blogs.sap.com/innovation/?p=1833697blog wave 2 thumbnail Jen Cohen Crompton – Digitalist MagazineAccording to a USA Today article, last year, 92 million people shopped on Black Friday while 131 million people shopped on Cyber Monday. Although this data shows a growing trend in online shopping (and it shows no signs of slowing), these stats show that some diehards still live for that Black Friday rush of physically pushing through stores, throwing elbows to keep their places in line, and the overall allure of those shiny Black Friday deals.

For those looking to chip away at the $ 804 that the average American shopper will spend on Christmas shopping (source: National Retail Federation), be sure you are well prepared for your shopping experience. Get ready by grabbing the basics: comfortable sneakers, some snacks and water, and your smartphone. And before you hit the streets and have to rely on data networks or wifi, preload that smartphone with these five essential apps that will make you feel like a Black Friday pro.

Deals!

Coupons.com.
 Although this app seems pretty basic, it’s the solution to the 3942834 loyalty key tags that you just had to remove from your keychain. This app features loyalty card linking and Passbook-integrated coupons, making it one of the most convenient and smartest ways to get instant discounts with a smartphone (especially the iPhone). The app offers online and in-store offers, grocery and retail deals, and GPS-enabled notifications, which will ultimately help you save a ton of money. Brands such as Toys R Us, JCPenny, and Dick’s Sporting Goods are all on board, so make sure you download it on the App Store or Google Play.

TGI Black Friday. This app is well organized and easy to navigate. With deals sorted by product type and store, you can create a personalized shopping list with all the items you are looking to purchase. Use the “My List” function to create your list then get shopping. When you redeem an awesome deal, you can also share via social media so others can bask in all your savings glory.

Getting there, getting around, and getting home

Waze. Unless you’re considering taking public transportation, there is a good chance you will run into that Black Friday traffic somewhere along your journey. To give you the best chances of arriving on time and without too much headache, use a navigation app such as, Waze. Waze is the world’s largest community-based traffic and navigation app that allows you to join other drivers in your area who “share real-time traffic and road info, saving everyone time and gas money.” The app not only shows the best route from here to there, but also will reroute when a major distraction is detected up ahead. Get it, use it. You’ll be happy you did!

Honk. Yes, there are times when we all walk out to the parking lot and realize that we have no idea where we parked our car. With our hands full and bags on our shoulders, the last thing anyone wants to do is walk around aimlessly scanning the parking lot and furiously clicking the key fob to locate the missing car. Instead, use Honk, which is a car finder app (that conveniently doubles as a meter alarm to avoid the holiday parking ticket damper). When you park, the app will drop a pin and take all the guesswork out of the search.

Sit or Squat. Gotta go, gotta go, gotta go right now? If you or your shopping partner needs to quickly find a restroom, there is no better option than the toilet paper maker app Sit or Squat (from Charmin). Available for Android and iPhones, iPads, and the iPod touch (requires iOS 4.3 or later), the free bathroom locator currently has more than 100,000 bathroom locations in its database and shows public restrooms near your location on a map or in a list. A bonus is that clean bathrooms are awarded a “sit” designation while the not-so-clean facilities are given a “squat” label. Help out the person who has got to go next by leaving your own review or adding new, undiscovered toilet locations.

Grab these apps and make this year’s Black Friday shopping experience the best yet – and of course, enjoy your holiday and be careful!

]]>http://www.digitalistmag.com/technologies/mobile-applications/2014/11/26/5-essential-black-friday-apps-01833697/feed11833697 http://www.digitalistmag.com/future-of-work/2014/11/25/hold-standing-meetingsliterally-01829758</link>
http://www.digitalistmag.com/future-of-work/2014/11/25/hold-standing-meetingsliterally-01829758#comments Tue, 25 Nov 2014 13:00:41 +0000http://blogs.sap.com/innovation/?p=1829758Most of us have seen it – the conference call spoof that we can all relate to in more than one way (I am often the person who gets cut off and has to dial back in, or I think everyone can hear me, but I am muted).

This Conference Call in Real Life spoof is a great example of edutainment – it’s entertaining while being educational as it brings to light a pressing real world issue: how to make meetings productive and relevant.

In the world of remote workers and the need for accountability, there are some meetings that are initiated by an organizer who doesn’t have a clear motive, agenda, or strategy – or format. Because of any of the stated reasons, these meetings can be complete fails.

But, because meetings will never be eliminated, we must work to figure out how to make them effective and how to minimize the meetings in the business world that often stray from the streamlined “what you need to know in a limited amount of time” to “let’s have a meeting about a meeting” – something we all dread. So instead of making these mandatory meetings ones where people “show up,” but aren’t really there (and often dread attending), consider a strategy embraced by some popular, fast-moving tech companies – standing meetings. Yes, these meetings are formatted so people are literally standing.

According to a Washington University in St. Louis study that used wearable sensors to measure participants’ activity levels, “standing during meetings boosts the excitement around creative group processes and reduces people’s tendencies to defend their turf.”

So, take a cue from these findings and consider standing meetings. They really do work because of these three factors:

  1. Standing meetings are short. Most people don’t enjoy standing for long periods of time, so meetings where everyone stands will be limited to a maximum of 15 minutes. Any meetings longer than15 minutes, will make most people want to sit. Knowing that the timeframe of a meeting is a strict 15 minutes or less (and the standing serves as a constant reminder) will allow the meeting organizer to stay on track and to keep the messages streamlined. Also, this could reduce the “dread” from attendees who would rather be doing something – anything – else.
  1. Standing stimulates paying attention. Go ahead and try it. Sit down at a meeting and then stand up. Immediately, when you stand, your body is activated and you become more alert and engaged. Standing requires work – and so does listening. So to ensure your attendees become more active participants in your meetings, make them stand and physically become engaged, which will result in higher mental engagement.
  1. Standing is associated with action (and energy). Consider sports teams who often hold pep talks and short meetings during breaks while standing. They generally don’t sit down and let the energy and excitement deflate from the group dynamic. Instead, the coach will encourage everyone to stand, huddle together, feed off the energy, then get right back to work. Keep the energy from meeting attendees flowing as they stand and prepare to take action.

Although standing meetings work best for status meetings and quick brainstorms as opposed to complete strategy or budget meetings, knowing the format that will work best for your content, having a clear goal, and inviting the right people to the meetings will always be the combination to organizing and hosting more effective meetings.

]]>http://www.digitalistmag.com/future-of-work/2014/11/25/hold-standing-meetingsliterally-01829758/feed21829758 http://www.digitalistmag.com/industries/travel-and-transportation/2014/11/21/week-tech-ibm-email-facebook-work-samsungs-milk-uber-01811910</link>
http://www.digitalistmag.com/industries/travel-and-transportation/2014/11/21/week-tech-ibm-email-facebook-work-samsungs-milk-uber-01811910#comments Fri, 21 Nov 2014 18:00:17 +0000http://blogs.sap.com/innovation/?p=1811910Friday Flashback11 Jen Cohen Crompton – Digitalist MagazineThis week in tech, we saw two popular tech companies fighting for attention at work and supporting better work collaboration through internal social networks. We also saw a new YouTube competitor brought you to by the Apple nemesis and a firestorm around the ethics and data abuse by popular social sharing transportation company, Uber.

Here’s what happened, this week in tech.

1. IBM’s Turn at E-mail. IBM is taking a shot at e-mail, following Google and Microsoft who each released their own new innovative e-mailing platforms. IBM Verse is an e-mail system for businesses, that uses collaborative calendars, to-do lists, and online documents. It also includes tools such as instant messaging, video chat, file sharing, and social updates. Using the new platform, IBM wants to help employees prioritize e-mails instead of aimlessly searching and sorting through e-mail chains. And speaking of better e-mail chains, IBM Verse uses a visual display of who is connected, how they are connected, and their titles within the organization (a little CRM added into the system). IBM says that this e-mail system learns over time, so if you’re missing an e-mail response or missing an important action item, IBM Verse can highlight who or what you might be missing. This e-mail system will start out as a freemium service, and is only aimed at businesses in the beginning stages.

2. Facebooking at Work? While Facebooking at work is something that is often prohibited, the popular social networking company wants to change that. Competing with sites such as LinkedIn, Facebook wants to put itself on your work to-do list. Creatively coined, Facebook at Work, the modified platform is meant for employee-to-employee communication in hopes of making it easier to collaborate on projects or plan that yearly holiday party. The goal is to create a separate space for each user so there is no correlation between a personal and work profile (whew!). Right now, the platform is being tested and Facebook’s 8,348 employees are successfully using Facebook to communicate internally and are using it as an alternative to e-mail. They are also using it as a project management tool, a way to share news, and they are even sharing pictures of broken equipment with the goal of getting it fixed. As of now, Facebook at Work is still in testing mode and is being implemented by some “unnamed companies,” but it will soon roll out to the rest of the corporate world. Get ready.

3. Milkin’ It. Samsung is launching Milk Video, which appears to be a competitor to YouTube. While the videos on the platform are currently exclusives for Galaxy devices, Samsung is curating videos from YouTube, Buzzfeed, Vice, College Humor, and Funny or Die, and the some sites are making some original content just for Milk Video. One of the biggest differences between Milk and YouTube is the absence of ads (although there is nothing mentioned yet about native ads or sponsored material). As of right now, Milk Video claims that the platform is not supported by ads nor will users see ads before videos, contrary to its YouTube counterpart. The Milk Video app is free, but only available to those using a Samsung Galaxy device. No word yet on whether Samsung is going to expand beyond these devices.

4. Uber Overdose. Uber might have outdone themselves this time – but not in a good way. After some not-so-great remarks made by CEO and founder, Travis Kalanick, and senior vice president Emil Michael, at what was intended to be an off-record lunch, the Uber path did not follow the GPS to success and the Uber folk definitely took a wrong turn. During the not-so-off-the-record conversation, Kalanick stated that $ 1 million was budgeted to find dirt on journalists who were critical of the company. The conversation included the intent on investigating the journalists through misusing their personal data and also mentioned they were targeting the journalists’ families. When Kalanick was confronted by another lunch attendee about the dangers of this practice, Michael replied that no one would know it was them. Wow.

And when it didn’t seem Uber could get their foot any deeper into their mouth, Kalanick made a very weak attempt at apologizing for Michael (which didn’t impress anyone), and published a blog post that outlined how Uber allowed employees to access rider data for “legitimate business purposes.” While Uber is trying to gain trust back from everyone, it looks like Lyft, Uber’s competitor, might be getting some new business in the very near future.

That’s it for this week in tech. Did we miss anything? Let us know!

]]>http://www.digitalistmag.com/industries/travel-and-transportation/2014/11/21/week-tech-ibm-email-facebook-work-samsungs-milk-uber-01811910/feed11811910 http://www.digitalistmag.com/innovation/2014/11/20/save-thanksgiving-turkey-mobile-apps-01801052</link>
http://www.digitalistmag.com/innovation/2014/11/20/save-thanksgiving-turkey-mobile-apps-01801052#respond Thu, 20 Nov 2014 13:30:05 +0000http://blogs.sap.com/innovation/?p=1801052iPhoneTurkey e1416450112766 Jen Cohen Crompton – Digitalist MagazineIf you’re new to cooking Thanksgiving dinner (or maybe you’re just trying to not burn the turkey this year), we have some interesting apps to help you through this hectic holiday dinner.

From getting your guests together to making the most delicious Thanksgiving meal ever, we have you covered!

Planning apps

Even if you’re a natural-born planner, you might want a better, more innovative way to plan. Here are two apps that can help get your guest and grocery lists organized.

  1. MealBoard – Making a grocery list can be painful on a normal day, but a grocery list for Thanksgiving can be a challenge for the most experienced planner. Here to help is the MealBoard app, priced at $ 3.99. Input recipes of your own, or a recipe from any cooking website, and this handy app will generate your grocery list for you. The app also supports major brand grocery stores, so you know how much you’ll be spending ahead of time.
  1. Paperless Post – If your Thanksgiving guest list has more people than fingers you can count on one hand, the Paperless Post app will be your hero (and it’s free!). It can send invites, track the RSVPs, and let you communicate with those attending your fantastic feast.

Cooking apps

After unloading the groceries and confirming your guests, it’s time to get into that turkey! Cooking multiple foods at once and having them simultaneously finished to perfection is about as challenging as herding cats. So, to help you out, here are a few apps to keep you from burning the turkey, making sure you have a synchronized ending, and that your guests’ mouths are watering (and their tastebuds aren’t rebelling) at the first bite of your delectable dinner

  1. Time to Roast – This app focuses on the turkey – or whatever else you plan on roasting. By selecting all your settings (temperature units, oven type, and preheating times) and your type of meat and the weight, the app will calculate the perfect cooking time and temperature. You can also select the time you want to have food on the table and the app will calculate a timeline for your turkey roast – from preheating the oven to serving.That will take the guesswork out of timing that turkey! Time to Roast is available for $ 1.99 in the iTunes store.
  1. Snapguide – While reading a recipe can get the job done, Snapguide gives you just a little more by showing a step-by-step visual guide for a variety of Thanksgiving recipes. This how-to app (which is free) is almost like having a personal chef in the kitchen showing you what to do next. From appetizers to desserts, Snapguide can get you through that tedious Thanksgiving prep. For an added bonus for those who are crafty (or try to be crafty), this app also gives visual presentations on DIY projects. Hello, post-turkey crafting with the kiddos!
  1. KitchenPad Timer – With many foods cooking simultaneously, it is easy to lose track of time for each burner, oven, or microwave (it’s totally okay if you use the  microwave – no judgement here). Having just one or two timers are not going to cut it – and that’s where KitchenPad Timer comes in. The app gives you a visual of burners and oven timers, which you can name individually, add designated temperatures, set different alarms for each, and of course, keep track of the time for each food item. At $ 1.99, this app will help you take control of your kitchen!

Activity apps

And how about those pre and post-turkey activities? While slaving away in the kitchen for hours, to produce what seems like only a few minutes of devouring the amazing food, is not the only perk of Thanksgiving day, there are a few apps that will help you enjoy the rest of the day with your company. Whether your activities of choice include watching football or seeing the amazing floats in the Macy’s Thanksgiving Day Parade, here are a few apps that can help you enjoy Thanksgiving day, pre and post-feast.

  1.  NFL Mobile – Football has become synonymous with Thanksgiving. If you can’t get to a television, or you’re just wondering what time the game starts, the free NFL Mobile app can tell you everything you need to know – stats, times, and highlights. You can also use the app to check in on your fantasy team because seriously, you need to know who you can tease during dessert if your team is stomping theirs. The app also provides streamed video and audio so you’re always in the action.
  1. Macy’s Thanksgiving Day Parade 2014 – For 88 years, the Macy’s Thanksgiving Day Parade is now a tradition for many. This year (as they did last year), Macy’s has released a free app that lets you in on the fun even if you can’t physically be at the parade. Use the app to see video clips of classic balloons and to let the kids create their own parade balloons. The app lets you become an insider of one of the largest Thanksgiving Parades.

And although we highly recommend taking some time on Thanksgiving to take a digital detox and enjoy your time with those around you, hopefully these apps will help ease the stress of the day and give you more time with friends and family, all while impressing them with your planning and cooking skills.

]]>http://www.digitalistmag.com/innovation/2014/11/20/save-thanksgiving-turkey-mobile-apps-01801052/feed01801052 http://www.digitalistmag.com/industries/high-tech/2014/11/14/week-tech-weather-service-hacked-sony-gets-race-facebook-gets-basic-01769986</link>
http://www.digitalistmag.com/industries/high-tech/2014/11/14/week-tech-weather-service-hacked-sony-gets-race-facebook-gets-basic-01769986#comments Fri, 14 Nov 2014 18:00:43 +0000http://blogs.sap.com/innovation/?p=1769986Friday Flashback11 Jen Cohen Crompton – Digitalist MagazineThis week in tech, the U.S. weather service had quite a scare, thanks to (you guessed it) a hack. On a positive note, Sony entered the race for streaming television services, Facebook gets back to basics, and a company created a solution to that annoying smoke detector beep indicating the battery is low.

Here’s what happened, this week in tech.

1. U.S Weather Service…Hacked. Although the forecast for the impending Polar Vortex is unfortunately a fact and not just a cruel joke played by hackers, The National Oceanic and Atmospheric Administration (NOAA) websites were recently hacked, which did cause some disruption. Four websites were affected and forced officials to shut down some of its services, which included cutting off satellite data, the National Ice Center website, and other related websites. While hacking the weather “did not prevent us from delivering forecasts to the public,” says NOAA spokesman Scott Smullen, the bigger scare is what may be hacked next.

The military, local government, and businesses rely on the U.S. weather service for accurate information, and it is considered a vital asset of the government. Overall, it appears that China was the culprit behind the hack, which happened in late September or early October, but officials were not aware until October 20th. NOAA told the public that there was “unscheduled maintenance” on their network, which has now resulted in a controversy over whether or not the NOAA should have immediately shared the truth with the public.

2. Sony Tackles Television. Another company is hoping to capitalize on the desire from consumers to deviate from standard cable and enjoy a new way of watching “television.”  Sony is entering the television streaming market with Playstation Vue, a cloud-based service with 75 channels. This streaming service will include content from Fox, NBCUniversal, CBS, and some other popular networks. Although Sony isn’t launching Playstation Vue until early 2015, invite only beta testing will start later this month. With competitors like Netflix and Hulu, Sony hopes to bring in the market with competitive pricing and different viewing options.

3. Back to Basics. If you’ve been confused by the recent Facebook articles about their privacy policies, don’t worry because the company is stepping in to clear up any misleading information.

Recently, Facebook has updated many of its policies and noting what users really want from their Facebook experience. As a result, Facebook issued an update and although there is no drastic change in their privacy policy, Facebook has rewritten the policy using a more basic language so users can actually understand the policy and decipher between what is real information and what is not. A page, appropriately labeled Privacy Basics, guides the user on Facebook’s policies. With 36 languages, it’s meant for users to easily understand how to manage what their friends and advertisers see about them. A step in the right direction for user privacy concerns? Probably, since ensuring users can actually understand a policy seems like it could clear up a lot of confusion.

4. A Smarter Smoke Detector. There’s nothing more annoying than that constant loud beeping of a smoke detector when the battery starts to run low. Yes, it is an important function of the device, but it is no doubt a nuisance. But, no fear because the company, Roost is offering a solution with a new battery that will change the way the device is connected to the home.

The smart smoke detector features a smart battery that connects into an existing smoke detector and is then connected to an app. The smart battery is a typical 9 volt with a low-power Wi-Fi chip and lasts up to five years, costing $ 35 per battery. The app detects the battery level and sends an alert weeks before the battery is about to die, therefore, eliminating the need for the incessant beeping (that is, as long as you still take the time to change it). The app also does more than just prevent that terrible beep –  it provides access to emergency numbers and sends push notifications and alerts when the alarm is activated. And there you have it – a smart smoke detector.

That’s all for tech this week. Did we miss anything? Let us know!

]]>http://www.digitalistmag.com/industries/high-tech/2014/11/14/week-tech-weather-service-hacked-sony-gets-race-facebook-gets-basic-01769986/feed11769986 http://www.digitalistmag.com/lob/sales-marketing/2014/11/13/find-the-hero-and-tell-the-story-01760050</link>
http://www.digitalistmag.com/lob/sales-marketing/2014/11/13/find-the-hero-and-tell-the-story-01760050#respond Thu, 13 Nov 2014 14:00:15 +0000http://blogs.sap.com/innovation/?p=1760050Rudy. Rocky Balboa. Braveheart. Lassie. Everyone loves a hero.hero in story e1415837359131 Jen Cohen Crompton – Digitalist Magazine

Through the evolution of storytelling, one thing has always remained – we, as the audience, love a good hero. The story of a seemingly unlikely character overcoming the odds and accomplishing something amazing and becoming a celebrated hero is one that resonates with many – we want the “good” guy to win.

In a narrative, the hero archetype is the character who faces a challenge and is able to overcome the challenge. The challenge could be finding success in the face of adversity, showing the world that the underdog always has a chance, or showing a connection to a human that is so deep that it causes miraculous things to happen. No matter the situation, the hero is loved.

When it comes to business and storytelling in marketing, a great heroic story can work. A story featuring heroism can grab attention and unleash that emotional connection we often try to create through the hero’s journey.

But, finding a hero in your business story isn’t always the easiest challenge to overcome. To do it, you have to be creative and think outside the box and below the surface of what appears to be the story. You also might have to push your brand into the backseat.

For instance, take the story of the 23-year-old Chicago Bulls basketball player who was featured in a YouTube documentary, The Return, which chronicled the terrible leg injury he endured while competing in a game. The six two-to-three minute webisodes showed his journey from the injury to being badly injured and his ongoing quest to hit the court as a competitor once again. During the clips, Rose shares his feelings and is candid with his viewers about what’s happening inside and out.

And his viewers become intrigued. They want Rose to get back on the court. They want him to become a “hero.” Oh and a popular footwear company sponsored the show – and the brand was seamlessly integrated into the clips, but was not a

Let’s block ads! (Why?)

Jen Cohen Crompton – Digitalist Magazine