Arms Traders Almanac – Social analytics offers a more honest read on opinion and sentiment

Big data is a term that was long ago co-opted and occasionally debased by technology and marketing departments. That probably explains why its original intent has been forgotten. The ‘big’ in big data referred not simply to volume but also to the plethora of sources and a major contributor to that was the rise of social data.

The ability to ‘understand’ an individual using his or her’s own free form expression via social channels has emerged as a key strategic skill for the modern data-driven corporation.

Sharing, and not really caring

One of the largely unique characteristics of social data is the speed and ease with which people share information on social channels. This makes the content less heavily curated, which in turn gives a more honest read on opinion and sentiment.

Also, while character limits make social data a little more difficult to analyse owing to the widespread use of abbreviations, the length and format limitations encourage people to cut to the chase and leave out the social niceties that can obscure meaning.

Making it a more potent barometer of sentiment than other more considered responses.

So what is unique about the insights provided by social data? And how does it help us to understand customers better or differently from traditional data sources?

According to Steve Lockwood, measurement & insight lead, Australia & NZ, Facebook, “Traditional data sources can be either expensive, inaccurate or biased, or a combination of each of these limitations. Social data across Facebook surfaces real interests, characteristics and behaviours of real people.”


(Image Steve Lockwood)

Lockwood says insights provided by social data, in conjunction with Facebook’s data partners, provided a more insightful and actionable view of people in Australia than any other communications channel.

Twitter data plays a very complementary role to traditional research methods. While traditional research firms use phones to survey customers, many younger generations don’t have a land line phone. In additional, traditional research firms often take weeks to compile research while Twitter data has the ability to offer instantaneous insights. Also, most importantly, Twitter data provides unsolicited feedback to brand managers so you often know exactly what the customer is thinking, unprompted.

Meanwhile, Inna Kolyshkina, head of the South Australian chapter of the Institute of Analytics Professionals of Australia (IAPA), argues that social media data can be harvested to quickly gain insight. This can then be used to proactively manage an organisational public image, brand awareness and customer satisfaction.

“A good example is real-time consumer opinion and sentiment analysis. Marketers have always needed to monitor media for information related to their brands – whether it’s for public relations activities, fraud violations or competitive intelligence.”

But fragmenting media and changing consumer behaviour have crippled traditional monitoring methods, she suggests.

“Market research can provide companies with a snapshot of how a small sample group perceives a product at a moment in time, but  social media data sentiment analysis is more akin to a continuous video.

“Word-of-mouth is now available on the web: opinions on anything are expressed in reviews, forums, consumer sites such as, trip and blogs for instance.”

Beyond identifying how consumers feel about brands or celebrities, opinion and sentiment analysis can predict market behaviour and sales volumes, according to Kolyshkina.

A better understanding of customers also makes it easier for brands to tailor their messaging.


“We know many advertisers are reliant on utilising Twitter data as part of their focus group process. Many advertisers will tweak or change their ads entirely based on feedback they’re hearing from Twitter,” a company spokesperson told the Almanac.

“Twitter data plays a pivotal role in helping companies understand what they like and don’t like about their products. We know a burger chain has made tweaks to the size of their french fry offerings and changed the music playing at their franchises. By listening to their customers unsolicited feedback, brands are much quicker to hear from their brands.”

There are a couple of common ways that brands are leveraging social channels to make their advertising more relevant. Firstly, if they can achieve a reasonable match rate to social platform members, a brand can very effectively leverage the information that social platforms hold about their members’ likes, dislikes and characteristics to deliver targeted ads through the platform’s in-stream advertising. Secondly, social channels can be a great way to test consumer reaction to a creative idea before committing to the time, cost and un-retractible visibility of an above the line campaign.

Kolyshkina says, “Analysing customer behaviour patterns online such as the sites they reach, social media they use, their interests as expressed on services like Twitter, Facebook, LinkedIn or Quora along with the purchases they make allows companies to understand the needs of the customers and display the advertising material that the customer is likely to respond to.”

In simple terms, if a customer’s behaviour suggests that the customer is interested in fashion, the ads they see may be about clothing.

Social of course also allows companies to hear directly back from their customers and prospects.Facebook’s Lockwood says, “Being focused on the needs of consumers is key, rather than blasting the same product messages to everyone. Advertisers that achieve good success on Facebook and Instagram recognise this.”

Zuckerberg’s Curse also offers a number of different targeting options.

Advertisers who create content to communicate a relevant message to the right audience at the right time generate the best return for their business, according to Lockwood. “Our platforms are pretty unique as communication channels, in that we can reach a broad audience in a segmented and personalised way.”

“The most successful partnerships we have are with advertisers who have a clear idea of what they want to achieve by advertising on our platforms – brand building, new customer acquisition, existing customer retention and growth and so on – and leverage our targeting capability to ensure their execution is in line with those objectives.”

Kolyshkina has her own examples. “Users of Amazon are encouraged to post reviews of the products that they purchase and little attempt is made by Amazon to restrict or limit the content of these reviews. The reviews provide accessible and plentiful data for analysis.

As a more specific example Kolyshkina refers to an analysis of social data on public sentiment towards the four major Australian banks in 2012.

This work delivered among other findings, the following insights:

The positive consumer feedback focused on helpfulness of staff in the branches.

The complaints focused on banks’ online interface deficiencies (particularly that of Bank Three), customer service in branches (most negative feeling being created by Bank Two), email spamming (particularly Bank One), ATM issues and customer fees (no difference across banks).

The high-value consumer groups that had markedly lower sentiment than the rest were young professionals who commented on Bank one’s online interfaces inadequacy and those having children who expressed disappointment by Bank two for failing to meet the recent RBA level of rate cuts.

High touch

Social channels can be useful as a medium to directly engage with customers to gauge their responses; equally they can be a powerful tool for discreetly polling customer opinion. Using social channels to create a dialogue with customers means a brand can effectively keep a finger on the pulse of how it is being perceived in the market, and quickly flag any issues before they hit the mainstream press.

Using social data to understand customers without direct interaction is something of an art, because it requires reasonably sophisticated semantic analysis to get true insight into the mind and mood of consumers. Brands that do invest in good analytic resources – and this is an area where a good analyst still trumps most automated systems – can reap substantial benefits by tracking how consumers speak about them, as well as to them.

Merging with non social and offline data

Increasingly social data is being used to extract even greater value when blended with offline and non social data.

Merging social data with other data streams can be straight forward or profoundly complex, in the same way that merging any other data source is.

If there is a linking key – an email address, name, phone number in common – matching is simple. Where blending data can be difficult is when you are dealing only with aggregates and reports from each source. In those instances, techniques like scaffolding data and finding proxy keys can help, but it remains as much of an art as a true science.

Data on the customer location can be easily obtained from the social sites. Then, depending of the type of insights brands are seeking, they can use customer location to overlap this data with various other inputs such as socioeconomic features of the area they are in for example, unemployment levels, average household income; or geographic characteristics such as weather and climate.

“We have established partnerships with the best data providers in the market, to enable targeting options using offline demographic and behavioural information in combination with online Facebook and Instagram data,” says Lockwood.

“Partner categories are available to every advertiser in Australia, and information such as socio-economic status, life stage, shopping habits etc can be used for targeting on our platforms. In addition to this, our Custom Audience product allows advertisers to leverage the information that they know about their own customers and website visitors to develop relevant messaging even further.”

Ownership of data

The exact nuances of who owns data and how they can use it does vary between social platforms, and it takes a close reading of each one’s privacy and data use policy in order to understand it. As such, brands should be very conscious of privacy and legal considerations when accessing or using social data.

Firstly, in the majority of cases, the right to use social data sits primarily with the platform provider, and the brand may be seen as a third party user, which carries various constraints on use.

Secondly, when actually using social channels to post, share and repost, brands should be just as conscious as individuals that once posted on a social platform, their data can be taken, repurposed and shared in a manner not under their control.

Once information is out there, it’s able to be pushed into the public domain even if posted privately in the first instance, so companies need to be very cognisant of potential fall out from content posted under their brand’s username.

Facebook owns its own data and is strict with how that data can be applied.

“We take privacy very seriously, and we don’t pass any personal information of people in the Facebook and Instagram community to any third-parties. The data behind our targeting capabilities can only be used when communicating to people on our platforms,” says Lockwood.

“We think ads should be as relevant and interesting as the other content you consume on Facebook, and with the launch of recent products, we’re providing advertisers the ability to reach customers and prospects in ways that are more relevant and customised to them.

“And we have built these experience in a way that protects our users’ privacy – neither advertisers nor third-parties get any personally identifiable user data from any of these products.”


So what  kind of software/tools and skills do companies need to make the most of their social data?

On the capabilities front to make the most of social data you need staff with experience across  computing science, data management, data science and analytics. And they don’t come cheaply.

Research into labour costs by IAPA revealed  that typical data science salaries in Australia can average $120,000 a year, but that figure rises as high as $200,000 when it comes to skills with social data.

It is not a problem unique to Australia. The McKinsey Global institute for instance predicts that by 2018, the US alone could lack 140,000 to 190,000 people with the qualifications to analyse data to effectively contribute to large business decisions.

The simpler way to imagine that is that for all intents and purposes those people don’t actually exist in the kinds of numbers that make them easy to find.

Worse, while you may be able to find data science practitioners, getting one who has both the capabilities of the discipline and knowledge of a specific market – say retail for instance – is very difficult in a market like Australia. That is why there is a growing trend towards analytics outsourcing.

Earlier this year Sri Annaswamy, founder and director of Sydney-based Swamy and Associates, told US-based “Analytics outsourcing is the fastest-growing part of the outsourcing industry. Annaswamy said the early impetus came from the US and the UK, and that cost alone was not the only driver. It’s about finding people with sufficient skills to deliver this on a large scale.”


(Image:  Sri Annaswamy)

In that same article Anna Frazzetto, senior vice president and managing director for international technology solutions at global executive recruiter Harvey Nash was reported as saying: “[The] offshoring of data science and analytics is a rapidly rising trend in the outsourcing and offshoring industry, and it’s a direct result of the push in recent years by businesses worldwide to collect, analyse, and make the very most of their big data.”

Fighting back

There are also local initiatives to address data skills shortage, of which social is just one aspect. Earlier this year, Melbourne Business School (MBS) and SAS Australia announced a three year collaboration to provide Masters of Business Analytics students with the advanced business skills and business analytics software. Companies such  Woolworth’s, AT Kearney, SEEK, Brightstar, Suncorp and Telstra lined up behind the program offering financial assistance with scholarships and access to data sets for study.

At the time of the announcement Emma Gray, the chief data and loyalty officer at Woolworth’s said, “Woolworth’s is passionate to further the science of developing better insights for better business decisions through advanced data analytics.”


(Image Emma Gray)


To really get into the juicy data for most social channels, companies need to access data through API feeds rather than just the front end ‘reports’ offered on their main sites. There are a lot of ETL (extract, transform, load) tools out there to make pulling data from various types of APIs into a database a simplistic and low-tech process.

To make use of the data, you need to have analysis software that allows at the very least some advanced statistical analysis as well as data codifying or manipulation. Ideally, text mining applications can take a lot of the leg work out of social analysis, but as mentioned previously, a top class analyst can easily derive a lot of insight without ‘point and click’ tools (and is likely to prefer to build their own solution anyway).

But it pays to be cautious of analysis tools that claim to be able to do full text sentiment analysis without a technical user or training/classification process. In reality this is a fairly complex and subtle analysis and should always in the first instance be done with human guidance.

A word of caution

Like all data,  social data comes with its own compromises and limitations. A 2013 study by Prince-
ton University’s Centre for Information Technology (CITP) called “Big Data, the pitfalls, methods and concepts for an emergent field,” by author Zeynep Tufekci, covered this in detail.

Tufekci a University of North Carolina professor and CITP fellow told Which-50 at the time that too many researchers treated Twitter as a “model organism” – something akin to the fruitfly in biology. (You can read more in MIT Sloan Management Review’s coverage of the paper)


(Image: Zeynep Tufekci)

At the time the paper was released it generated some heat in the social research community. Tufekci said “Twitter and all platforms have specific affordances – behaviours they reward and behaviours they discourage at the level of infrastructure – as well as site specific social norms that emerge overtime.”

She argued that an over-reliance on Twitter as a “big data” source could lead to mistakes where researchers over generalised from the data.

“For example, many studies of influence use Twitter as an example and conflate ‘Retweeting’ with influencing. A user, however, may retweet for a variety of reasons besides being influenced, including to make fun of or disagree with a tweet.

“Twitter’s specific affordances that make retweeting easy, as well as social norms (ie retweeting as a common behaviour) can lead people to overestimate the level of influence on social networks.

“It is a bit like studying fruit flies and then generalising to larger creatures – you cannot. Fruit flies were chosen (as model organisms) because they were small and fit easily into the laboratory.”

As was also reported at the time, Tufekci said the reason Twitter is used disproportionately in large scale big data research, especially those projects involving millions or billions of data points, was not always related to its efficacy as a source of  data for accurate analysis. Instead, she argued it was more about the Twitter data’s availability, tools availability and popularity, and ease of analysis.

She noted that while Facebook was (and remains) the largest social media platform, there is less truly public data on Facebook “and thus Facebook is less accessible by scraping or via Facebook’s API as many more Facebook users (estimated to be more than 50 per cent) have taken their profiles private compared with Twitter users (estimated to be less than 10 per cent).”

She also argued that the Twitter stream was relatively easy to access through widely available and popular methods (the Twitter Firehose, the spritzer, white-listed accounts, etc) compared to Facebook’s API which she said at the time was lesser-known with fewer ready-made tools.



Previous post

Getting into the mind of the customer

Next post

Mobile set to secure retailers pre-christmas sales: research