Imagine you see a rug you like at a café – not just any rug, but one that would put the perfect finishing touch on your living room… or as The Dude would put it, one that would “really tie the room together”.

A magnificent rug indeed.

You ask the employees if they know where it was purchased, but alas they have no idea. You can’t find any distinguishing brand, marks or labels, but you just have to know where you can get one. You’re now faced with the task of trying to distil the item down into a combination of words that might help pluck it out of obscurity among the billions of pages across the internet.

This process of adding and deleting specific words from searches is something we’re all used to… but what if there was another way?

What if you could take a photo of the rug, plug it into a search engine, and not only find it but also be given suggestions of other complementary furnishings from a database of interior designers, influencers and tastemakers?

According to ViSenze, an AI-led Visual Commerce company, visual search can outperform text-based search for these types of queries. In an experiment dubbed “The Dress Challenge”, human searchers were pitted against the AI to try to find a certain dress online; 96.6% of human searchers gave up before even finding the dress, while the 3.4% who stuck it out took 4-6 minutes to complete the challenge. The AI, on the other hand, found the dress in just 1/9th the time.

How Visual Search learns and improves

Visual Search engines are a departure from traditional text-based and reverse image-driven search queries. Instead they use an image as the search query and derive a result by replicating the human visual search process through AI neural networks and deep learning.

I use the term “replicate” loosely, as when you start to break down how the brain discerns a child’s crude drawing of a cat as an actual cat, it’s difficult to translate this into something that machines can understand.

The artificial neural network (or model) has to go through an identification and learning process each time it receives an image query (or input), tweaking the neurons that aid in the decision-making process (or output). If it returns a dog instead of a cat, it’s shown a picture of a cat that it can learn from for the next time the image is called upon.

The Visual Search race is on

2018 was an exciting year for computer-vision-based AI. Facebook introduced Face Recognition to the world, an ability to analyse the uniqueness of each human face and identify untagged photos of its users across their database.

Google Lens can now recognise over 1 billion items thanks to its neural networks having unfettered access to the entirety of Google’s knowledge base and consumer data. Snapchat also launched a Visual Search tool that lets users search images and return products on Amazon without leaving the app.

With major global brands like Ikea, Amazon, Target, West Elm, Asos and Uniqlo all having integrated visual search into their ecommerce sites and apps, and a plethora of platforms available from majors like Google and Ebay to notable start-ups like Slyce and Syte AI, how should we as marketers start thinking about visual search and AI driven visual learning in 2019?

Keep an eye on Pinterest

Pinterest’s first foray into Visual Search was back in 2015. Since then they’ve ambitiously backed the technology and continue to lead in this space. Pinterest teamed up with ShopStyle in 2017 to release Shop the Look, a new way for users to shop for products within a Pin. These Pins are powered by the ‘ShopStyle Collective’, an affiliate network that rewards influencers creating content fit for visual search. Pinterest’s retailer shopping product feeds utilise a simple tagging tool that makes Shop the Look a robust platform.

Pinterest Lens, a visual search tool accessed via the camera within the Pinterest App, boasts 600 million visual searches every month, owning the lion’s share of an estimated 1 billion global visual searches monthly. Pinterest’s visual search across Lens grew YoY by 140% between 2017 and 2018!

While there are a number of hurdles that need to be traversed in the lead up to paid visual search (attribution and reporting for starters), Pinterest is all-systems-go and a strong contender for a shot at the title against the traditional search behemoths.

Identifying new trends and influencers

Extensive research and social listening goes into identifying an influencer who’s relevant to your product, brand-safe and has an authentic voice with extensive reach amongst their social peers.

The same goes for identifying natural brand synergies with real life moments in the search for new trends and marketing opportunities. Until recently, the tools that assisted in this task had been similar to traditional search, scraped and distilled text-based interactions from across the web.

Talkwalker provides Visual Social Listening via their machine learning AI powered image recognition tools to identify major retail brands, and their attached influencers without a text reference to the product in the post. These “hidden mentions” are brought to the surface through their deep learning technologies’ ability to recognise 30,000 brands and logos in images across social media.


Interest-based targeting will become even more tailored as unique image elements provide marketers with the opportunity for more personalised shopping experiences. A great example of how this is coming to life now is the company Intelistyle. They offer an AI Personal Stylist that can be customised for a retailer’s brand and offer style suggestions based on uploaded images.

This AI can even offer visually similar alternatives to items that are out of stock or suggest fashion pairings based on your existing wardrobe. Intelistyle’s Head of Business Development, Sophie Burrowes, describes their AI technology as ‘like Spotify for Fashion’ where the more you engage with it, the more it learns about your style.

Rich media experiences

The magnitude of our visual engagement with the world is hard to quantify. The human brain processes images 60,000 times faster than text and 90% of information transmitted to the brain is visual.

Designing striking, visually driven media that’s fit for platform is particularly important at a time where consumption habits are shared across a number of channels and platforms. Machine learning may offer dynamic ads that not only learn and target from images across social channels, but also create a visually engaging media experience augmented in the real world via a Lens.

Ethical questions and “creepy” data

Considering GDPR and the rising concern around consumer privacy, marketers will need to define their approach to visually indexed information, especially when it informs targeting.

While I’ve mentioned some AI models can recognise brands, logos and fashions, there are countless others learning to identify other visual attributes. For example, Talkwalker’s AI doesn’t just learn about brands; its AI image-based applications also include scenery, objects, gender and age detection. Other companies now also offer untrained models that you can start adding your unique product images to.

A retailer’s ability to target based on detailed visual product attributes may seem more creepy than categorising behaviour based on purchases and categories. Those with pimples could be targeted with skin cream ads. Those who have had the same model car in their photos for the last five years might receive an offer to trade in for a newer model. Is this too much?

Marketers already have access to a certain level of “creepy” data – a balance somewhere between broad generalisations and over-personalisation. Visual search advertising and visual learning will likely compound this equation and usher in a new paradigm around visual-learned data and how it’s kept private.

So next time you see something you like while out and about, keep in mind that your smartphone camera might be the answer to the information you seek. You might just find that perfect rug to really tie the room together.

Nick Hayes

Nick Hayes is a Digital Strategist at iProspect Melbourne. With extensive experience across sports and membership-based organisations, Nick is an accomplished marketer with both the technical know-how and the strategic mindset to plan and execute campaigns large and small.