On the Small Data in Big Data

In Computing Now:  Video.  Small Data in Big Data — Ayse Bener, Ryerson University.   From data to knowledge, and leveraging the knowledge to decision. " ...  As data gets big and complex, there is a need for multiscale approaches in transforming data to knowledge. This transformation takes place by joining disciplines to model, analyze, learn, and extract knowledge in diverse domains from biomedicine to environment, engineering, business, software, and banking. This talk will primarily touch upon the relevance of small data in big data, the process (from dataset to the algorithm), interpreting and finding meaning to data, and algorithms to predict and cause the future events.  ... " 

Enterprise and Big Data

In Readwrite:  Well thought out piece. Have seen this same behavior in many hype cycles.  Large enterprises, who can afford it, accept the use of technologies like Big Data.  The emphasis should be on business need, but that is not sexy enough to get the attention of innovation money.   But it should.  Solve the problems, not the need for hype adoption.

Big Data Strategy and Hype

Forrester on Big Data Strategy.  A useful cautionary view on the floods of hype.  " ...  Don't Have A Big Data Strategy Yet? Good. ... Big data noise has reached the point where most are reaching for the ear plugs. ... " 

Sensors as Teenagers

In the Cisco Blog:  An excellent piece by Bill Franks of Teradata.  See his book I reviewed here: Taming the Big Data Tidal Wave.  About the Internet of the things and how it will operate, and about the information it will generate.   Some excellent points, good read.  " ... The primary issue for you to consider isn’t the way that our things will communicate directly to us the important facts they find. Rather, it is how they will communicate to each other as they identify those important facts. The connected things may very well end up being much more like a gaggle of teenagers than anything else. By that, I mean that they may chatter on incessantly about topics that few others find remotely interesting or important. Just as we grownups filter out the giggling teens at the mall, we will also have to filter out much of the chatter generated by our things. ... " 

Value of Your Big Data and Packaging it

What is the value of your big data?  Have talked to two startups recently that are addressing that. The promise of big data is to be able to leverage your own data with analytics, but also package and sell it for the use of others.  That packaging could include analytical preparation. Using something as simple as focused visualization for an industry.   More in ClickZ about the value angle.

Big Data and IBM

Not unexpected view and direction by IBM.  Riding the hype wave, with their own considerable analytics resources.  Those put together can provide a formidable capability:

" ... IBM wants to help IT managers apply company policies to their big data analysis projects. The company will be introducing new products and features to help organizations manage their new big data systems with the same rigor that they manage other IT operations, said Bob Picciano, general manager of IBM information management. .... 

IBM will add new features to its InfoSphere line of information integration and management software. It has also announced the general release of PureData System for Hadoop, a system configured for running Hadoop workloads.... " 

Interview on Big Data and Analytics for MidMarket

Friend Paul Gillin runs a Big Data Google Plus Hangout on the use of Big Data methods for Midmarket and beyond.  The 37 minute stream brings together current users of these ideas.    As a practitioner myself it has always been intriguing to see how this is now playing out at all levels of use.  In the past it was companies like Procter & Gamble and IBM, who could make these solutions work, now the possibilities have increased considerably.

The interview brings together three practitioners  of the use of analytics for improving business. Two areas of often mid market applications are discussed.  Biometric data in hospitals and marketing data to improve sales operations.

I mostly deal with is commonly called advanced analytics, so it was good to see the examples discussed here that use relatively simple, often just visual methods to get to solutions.   And emphasize the understanding of how companies make decisions.   Again I emphasize starting with the most simple methods.

Another topic discussed was how the Internet of Things is starting to create large amounts of data, in particular in health systems, and leading to big and complex data to be mined and leveraged.  In the case of hospital systems, the Internet of Things can include patients, monitors and diagnostic devices and a complex array of data types including imagery and quantitative measures.   Part of this system would be unstructured data such as text, such as doctor's observations that need to be assembled and attached to more structural data.

Once the data has been captured and simply presented  you can start to consider specific mining techniques to look for deeper points of value.  Every business has value opportunities.

In the final part emphasis is made of getting away from the need for data specialists.   Instead using tailored systems for particular industries, and then selling the analysis as a service.  Even suggesting that intelligence systems like Watson could drive the analysis,

Good piece to listen to to see what is happening in the field today that is important to the small and mid market company.  With pointers to the future.    This blog also covers the topic.

This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines of a smarter planet. I’ve been compensated to contribute to this program, but the opinions expressed in this post are my own and don't necessarily represent IBM's positions, strategies or opinions.  #MidsizeIBM

Small Data Becoming Big Data

In SmartData Collective, By Mark van Rijmenam
" ... small data can become big data by cleverly combining various data sets with different data formats; Combine weather data with your restaurant’s sales data to discover the impact of rain on your items sold and as such adjust your purchasing behaviour. Combine your customer data with their sentiment online to surprise them and create long-lasting relationships. Track how your customers behave through your shop and combine it with your sales data to see how you can adjust and improve your floor plan. Or combine online sales data with offline customer profiles to see how you can optimize your multi-channel approach for your small retail shop.... " 

My comments:  Interesting, but I dislike the name 'small data' as much as I disliked 'big data' before it ascended the hype ladder.  Data is useful if it is the right combination of private, public, large, small, stable or volatile.  It is useful if it solves a decision problem that improves something.  Lets quit pasting names on things.  Lets work on delivering specific value.

Trusting a Data Scientist

A Challenge: Just because a study contains numbers or big data or statistics does not mean it is correct. So you should not completely trust any scientist for business purpose.  Results from science should be carefully applied and validated to decision and business.  For that you need to know the business well enough to make the call of its correctness or value.

Unified Environments for Big Data Analytics

A good piece by correspondent Bill Franks, whose book on the topic I previously reviewed.  It addresses some problems I am addressing in a project design  now.

" ... It is challenging to make big data simple to access and easy to analyze. While there are many reasons for this, the one I want to focus on here is that handling big data, given how big data projects are usually implemented today, requires users to learn new tools and technologies. This makes adoption a difficult and lengthy journey. ... We are reaching a crossroads in the world of analytics. Over the years, we have seen analytic environments and data environments begin to merge through the advent of in-database processing within relational database engines. This trend had been leading to a world where analytic professionals don’t have to worry about moving data or accessing different systems to perform analytics. They could simply run their analytic processes against the data where it sits using the tools they know best.  Scale could be added to analytic processes even while also simplifying them and using less system resources. ... " 

How Big Data is Reshaping Marketing

In the coming weeks I will be passing along some of the excellent work by correspondent Phil Hendrix.   Here is a set of slides on How Big Data is Reshaping Marketing presented at Emory University, July 29, 2013.

Philip E. Hendrix, Ph.D. Director, immr and GigaOm Research analyst
phil.hendrix@immr.org  |  770.612.1488 (O)  | 678.294.8017 (M)  |  www.immr.org | Twitter: www.twitter.com/phil_hendrix

New Age of Algorithms

A good view of Big Data and Analytics algorithms in the CSM.  I like this article because it includes a number of real life application of analytics, several I had not heard of.  It does incorrectly imply that all this is new.  Analytics have been use for many years before data became 'big'.   Algorithms are simply applicable rules derived from analytical methods.   These methods use many forms of available data.

Understanding Analytical Databases

Interesting piece by Wayne Eckerson.  Linking to an upcoming talk.   How to get data to advanced analytical methods effectively?  The world has changed from the relational database of the near past:  " .... Today, these so-called analytical databases, or analytical platforms, span a range of technology from appliances and columnar databases to shared nothing, massively parallel processing databases, with unique extensions which in some cases include a MapReduce processing framework for advanced analytics. The common thread among them is that most are read-only environments that deliver exceptional price/performance compared with general-purpose relational databases originally designed to run transaction processing applications. ... " 

Advanced Maths Win Wall Street

A view of what used to be called 'rocket scientists' in Wall Street.  I recall going to the Santa Fe Institute and hearing a talk about this phenomenon,  just before the crash.   Now its math, big data and analytics.  Its not that I do not believe in these methods, I just look at the predictive value with caution.  Note something new here, the use of forward-looking, unstructured data analysis to look for causal triggers.    As opposed to quant forecasting.

Big Data for the Masses

Can everyone be a data scientist?  You will at very least need the strong support of data scientists. Similar to delivering analytics to everyone.  Its not so much about the technology to extract data from large and volatile sources, but about automating basic analysis and guiding the user to insights.   Not too much different from efforts by Tableau to do the same thing.  But taking this beyond just visualization analytics.  Insights that are useful to their business. A startup at Stanford is trying to deliver this.   Following. " ... 'Through powerful analytic models developed over a decade at Stanford, Ayasdi lets you find the needle in a haystack you didn't know was there, quickly.” Jonathan Ballon, GE’s chief strategy officer, said in a statement. “For GE and our customers with vast amounts of industrial data, Ayasdi's technology will be a powerful tool for predictive analytic models that can drive billions of productivity and efficiency savings."  ... '

Complimentary Webinar on Today's Marketing Strategies

Of interest.  by the SNCR which I worked with since its founding.  I plan to attend.

Complimentary Webinar 

CMO 2.0: Marketing Strategies for Dealing with Today’s Marketing Challenges
Tuesday, August 13, 2013 
11:00 AM - 12:00 PM PDT / 2:00 PM - 3:00 PM EDT

CMO 2.0: Marketing Strategies for Dealing with Today’s Marketing Challenges

Join the Society for New Communications Research and Francois Gossieaux, Senior Fellow of the Society for New Communications Research and Co-founder of Human 1.0, and Grant Johnson, Chief Marketing Officer of SDL for a webinar on “CMO 2.0: Marketing Strategies for Dealing with Today’s Marketing Challenges.” In today's social, mobile, big data and cloud world, significantly increasing demands have been placed on marketing. But if you're up for the challenge, there's never been a better time to be in marketing. This informative webinar will examine different challenges that marketing is faced today and will provide advice and tips on how to operate in a networked way. Learn the tools and strategies needed to successful market in today’s digital world.

Big Data Theory and Analytics

Big Data Analytics at Microsoft Research.   From a recent workshop there. " ... “The top-level takeaway for attendees was that big-data analytics is an area where important innovations can happen by a joint effort of the theory and systems community,” Vojnovic says. “It was appreciated that there is a need for developing suitable abstractions both in analyzing important theoretical problems, as well on the side of computation and programming. ...  “The event reconfirmed my belief that impactful research and innovation would result from a marriage of systems and theory. The event turned out to be a great success, and I am looking forward to new editions.” ... ' 

Retail and Big Data

A case of retailer A&P using a big Data Portal.  " ... This cloud-based portal enables retailers to share their proprietary data – including point of sale (POS), inventory, loyalty and more – with their supplier partners. It allows a retailer's vendors to work with and analyze the same data the retailer does in order to collaborate on mutual business goals. ... " 

Reliability of Big Data Predictions

In Knowledge Core:  Intriguing piece.  Philosophically oriented. A question we have asked of all analytics for many decades.  With the usual cautions and prescriptions.  But I ask:  Has this changed because you have more or more volatile data?  We always sought more data, and often did not have it.  Now we have it, and what does that mean?  Better predictions?  Or more difficult ones to explain?

Beginners Guide to R

Have not looked at this yet, but seems to be a good starter piece.  More people are asking me about R these days.  Short, simple, non technical.  " ... R is hot. Whether measured by more than 4,400 add-on packages, the 18,000+ members of LinkedIn's R group or the close to 80 R Meetup groups currently in existence, there can be little doubt that interest in the R statistics language, especially for data analysis, is soaring. ... "

Also, see my review of this book.  I have been using it as an introduction.