Data Mining the Social Media – The New Gold Rush

Our mobile connectivity, social media and hunger for apps have opened up the Pandora’s box of privacy breaches and their potential abuse. Not a week goes by when we don’t hear yet another story involving a breach by not just small app makers but giants like Google, Facebook or Apple. Government is weighing in, announcing a proposal for “Privacy Bill of Rights” or winning agreements with mobile leaders to enforce do-not-track or other privacy protection options.

Interestingly however, a recent study shows that users aren’t that concerned with privacy implications. For instance, according to a new study by Sociable Labs, more than half of the consumers (56%) regularly grant online retailers permission to use their Facebook data that range from access to basic profile like gender and name to details like birthday, product preferences and authorization to post status updates on Facebook walls.

Access to such data is proving to be a gold mine for the eCommerce industry. Online monitoring of  Social Media sites like Twitter, Facebook, Google+ and Youtube is a new emerging tool for anyone fishing for trends with your personal data. 


The story is particularly interesting with Twitter. The chatter of millions of tweets is increasingly viewed as a potential indicator of society’s mood. In fact, some wall street funds and traders are text mining and analyzing the words in tweets as serious “crowd sourced” market intelligence. A 2010 study by Indiana University found strong correlation between the collective public mood and the Dow Jones Industrial Average. At a company specific level, a similar study by Pace University found that the popularity of product brands, as measured by Facebook “Likes” or Twitter followers, can be a lead indicator of their individual stock performance.

Next time you tweet, remember that many web analytics firms are collectively trawling and monitoring your stream too.


Retailers and advertisers are jumping into this fray as well. They are quickly finding out that experience shared in social networks about any product also has the potential to predict what consumers or their network buddies could potentially buy. Isn't this a great opportunity to influence future purchase decision through targeted ads!!  Knowing what people like or want, as evident from their tweets or Facebook wall “likes”, can also help retailers better manage their supply chain. In the past such information was proprietary and only available from transactional data through companies such as Visa or Mastercard.  But, thanks to consumers’ sharing their opinions and personal data, often complete with where they are, a new gold rush for market analytics and targeted ads has just begun.


Walmart, for example, has set up a digital analytics division @WalmartLabs which reportedly uses crowdsourcing techniques to determine which items the company should stock based on its tracking of popular social media sites including Facebook, Twitter and Google+. 

Where do we go from here? Will there be new applications of social media mining for common good? Or will it evolve to be primarily a tool for tracking and advertising? Will there be consumer backlash? Or will consumers’ unwillingness to share personal details always be over-ridden by the instant gratification of getting the next discount or the coupon? 


Please comment here or simply share your thought by blowing in a new tweet to the cyber-space!

Does Internet limit our deeper thinking abilities?

In a recent conference, one fellow attendee, who happened to run a navigation software business, was lamenting how in-car navigation and PNDs (Personal Navigation Device) have atrophied our sense of direction and location.

He clearly has a point, but isn’t that true of all new technology? When human society transitioned to locomotive train and then to cars, we had to sacrifice our horse riding skills in favor of driving. When technology makes old skills useless, it often also requires us to learn new skills. The acute sense of space and direction of a traveler of pre-cartography days got replaced by our ability to decipher complex visual information of a map. And then the map reading skills got replaced by the new device-savvy awareness of running navigation apps. We give up certain skills but perhaps become smarter in some other ways as we adopt new technology.

While this premise of technology 'taketh some skill and giveth some skill' may often be true, clearly there are trade-offs. The point is eloquently highlighted in the context of the Internet by a 2010 book by Nicholas Carr – “The Shallows” – where he builds on his 2008 article titled “IsGoogle making us stupid”. Delving in to recent developments in neuro-science, he makes a case that the internet, while generous in its access to dynamic knowledge, encourages “rapid, distracted sampling” of information potentially atrophying our capacity for contemplation and deep reflection. In contrast, the age old printed book, with all its limitations, serves to focus our attention, potentially promoting sustained creative thought.  The internet while democratizing our instant access to knowledge has also added to our sense of distraction as we nibble at news, blogs, podcasts, videos while surfing from link to link to get quick answers.

Mr. Carr ultimately wonders if our over-exposure to online content is re-wiring our brains, making us shallow, addicted to quick bites of information, freeing us from the need to devote sustained attention to thinking as we constantly dip in and out of online content. Along with our power to get any content right away comes the inherent distraction that Carr argues may be changing our capacity to focus. The distraction is further accentuated by the fact that a single web page contains many different chunks of media – texts, audio, video, widgets, hyperlinks, advertisements – creating a “cacophony of stimuli”.

In a world of abundant content, ubiquitous connectivity and dynamic access, distraction is perhaps inevitable as we have the easy luxury of navigating content with the flash of a click.  But how this influences the style and depth of our thinking and adds to our cognitive load is the crux of Carr’s concern. Reading print material in many ways is a linear process allowing more stable attention, unlike the navigation of networked, dynamic content, peppered with hyperlinks, widgets and advertisements, which according to Carr constantly adds an element of being “distracted from distraction by distraction” (using the words from T.S.Elliot’s first of “Four Quartet”). Add to this the fact that our brains are not wired for multi-tasking (see my blog “Efficient Multitasking – the myth..” March27, 2011) and the challenge of fast cyber-hopping to our cognition and our capacity for deep thinking gets even bigger.

In today’s world of fast tweets, real time analysis and instant gratification, Carr’s concerns deserve attention as we grapple with the fire hose of everyday information.

Or, perhaps on a lighter note, all these may just be part of our evolution and at some time in the deep future, our brains will re-wire the neural connections, handle our online information without losing our capacity for deeper dive and focused contemplation.  After all, as Carr himself quotes the 17^th century Spanish dramatist Lope de Vega, some of the same concerns existed 400 years back too about proliferation of printed books -

“So many books – so much confusion

All around us an ocean of print

And most of it covered in froth”

Next Wave of Social Media - Will They be more Privacy Friendly?

Facebook is the biggest marketplace in the world where people gather, chat, share their personal data, play games and where advertisers have access to close to a billion people on the earth. Whatever may have been  the original intent or business plan, Facebook sure has become a dream-come-true for marketers, merchandisers and whoever else needs instant reach to a huge volume of consumers and their tastes, preferences and habits. It’s not a surprise that the impending public launch of this social networking platform is promising to be the biggest IPO ever – bigger than even the giants like Google, Apple and Amazon. It’s all about the lure of the biggest marketplace…..

So everybody wins. To the grandma keeping in touch with her far-flung grand-kids or to the boomer who just linked up with his or her high-school friends, to the high-school teen sharing the latest vacation pictures, Facebook is priceless; to the marketers it is a gold mine, and to the company shareholders, the $100Bn valuation may only be the beginning.

The insane money-making potential aside, will Facebook be able to extend its innovation to something bigger or should we be looking elsewhere for the next wave in Social media?

With our pervasive GPS tools, we all are increasingly fascinated by the location-based connections – not just with people but with places we visit and with things we do, We want to know where our friends are, if any of them is in the neighborhood and what's around us (without having to mess with any unwieldy paper maps). We want to geo-tag everything, geo-fence some of our connections and play geo-caching games… (I suggest Wikipedia if these were not in your current vocabulary!) There are even GPS dating apps – for your romantic networking, e.g., SmartDating app whose ‘People near me’ feature pulls up all the users (potential dates?) that are in your area! If SoLoMo (Social-Local-Mobile) is the new paradigm, will Facebook be still the reigning king or will it be up to the Twitters, Foursquares, Instagrams of the world to claim the next stake?

Whatever is the next wave, one thing is sure, social media will always need and thrive on our personal data. And that always brings up the dark side of our dependence and exuberance with social media. Should we be concerned with the recent disclosures that Twitter and other social networking companies are reaching into people's smartphones and retrieving their personal contact information without getting explicit permission? Twitter acknowledged that its "Find Friends" feature on smartphones (Android and iPhone) was merrily sending phone number and emails found in the address book to its back-office without clearly alerting the consumer.

Stories like this have a familiar ring … Every now and then a report of privacy breach leaks out, the company concerned apologizes, calls it a mistake and everybody moves on until the next story leaks out. Google, Apple, Facebook – all have been there. It all goes back to the premise that, benefits aside, all social media apps and platforms are a gold mine for marketers – where there are too many consumers ready to be tracked and lured.

Social media apps have brought people together, added many conveniences in our daily life and made the world a better place in many senses and I look forward to seeing continued innovation. We all however need to be aware of that its commercial value lies in reaching out to its captive base of the users and in knowing their tastes, preferences and whereabouts. May be, someday we all will ride out the hype-cycle of social media, or be just fatigued by our own over-exposure, find the sweet-spot of moderation and re-learn the pleasure self-reflection in our privacy.

Until then, Happy Tweeting. Look forward to you next Face-book update