Musings on the State of the Art

I’ve been thinking a lot about what e-commerce will look like in 10 years, or even half that time…

E-commerce as it stands today isn’t some new breakthrough technology-enabled process that was never possible before. Its still pretty similar to the act of what buying and selling looked like forever.

Customer wants something, customer goes to store, customer adds to cart, customer pays for it, customer checks out and leaves.

Note, the above could apply to a brick and mortar, or to an e-commerce player (such as my current employer – Quidsi Inc)

So what is different about online vs offline retail.

i) Unlimited shelf space.

Positive :

The constraints of what you can and cannot show customers is different in an online store vs a brick and mortar store. You can theoretically have a limitless collection of items, and are not constrained by the limits of shelf space in your store. No matter how large a store it is (think Costco) this is a very real limitation in the brick and mortar space. E-tailers are only subject to what inventory they can afford to purchase – but even that there are ways around (consignment, drop shipping, third party fulfillment, affiliate relationships etc.)

Negative :

Shelf space is a scarce resource and like all good scarce resources, brick and mortar stores monetize it. This is harder in an e-commerce world, and so e-tailers create scarcity in order to have an asset that they can monetize. (Landing pages, brand pages, micro stores within stores, the images on the home screen etc.) Also, the more products you add, the harder it gets to discover the right product for the customer.

ii) Supply chain

Positive :
E-tailers no longer need to manage and deliver product to stores across the nation. The complexities involved in keeping track of inventory per store, and its delivery is not trivial.

Negative :
The e-tailers cannot guarantee the same delivery experience across the country to all its customers, especially if it has only a limited number of warehouses. Also they cannot deliver the same immediacy that a physical store on your block can. Want something fast – no matter how quick an e-tailer can ship it to you – they cannot beat walking down the street and buying it in the corner store.

iii) Shopping Experience

Positive :
E-tailers can guarantee the same shopping experience to almost every customer that wanders into their URL.

Negative :
That experience has no local flavor to it, or for that matter any real personal interaction. Its a very impersonal shopping experience. No sales people, no friendly clerk to help you understand what you are purchasing etc.

I could go on, but you get the idea – in every way that e-commerce differs from traditional retail, there are opportunities for companies to be created and to grow. Here’s a few examples.

Discovery – Several large companies already exist to help customers discover the right product whether by search, navigation, merchandising etc. Endeca, Omniture, and a whole host of new start ups.
Delivery – E-tailers are competing nationwide to deliver fast free shipping and comapnies like webvan, freshdirect show that it is possible to take what would have seemed impossible only a short while ago and transform it into an online activity. Others will follow suit and the concept of local delivery for fast moving goods will become commonplace especially in urban areas. While this doesn’t exist yet, I think e-tailers (on the backs of delivery partners like UPS and Fedex) will also eventually allow you to pick a time-slot you want your package delivered (like Freshdirect allows now).
Personalization – Baynote, Certona, MyBuys, and a host of others are in the business of attempting to make the shopping experience more personal by showing you products, content, and experiences that they think are relevant to you at the appropriate time.
While all these are great, and the innovation and progress in this field does exist – I can;t help but wonder if this is what e-commerce will look like in 5 or 10 years.

However, several things have changed online, and they do allow for behaviors that were rarely possible before.

i) Access to information:
Information arbitrage is very difficult in a massively connected world.

ii) Access to similar people:
The ability to access a group of people that thinks and behaves like you is much more possible in today’s world than it has ever been before.

Why did I pick those two pieces to highlight? There are several other things that have changed, and are empowered by technology. However these two are going to do something interesting

Access to information includes price information. Eventually, customers will have total visibility across all products and categories. This will eventually drive prices to the lowest possible price as competition increases, and e-tailers will have to compete on something other than price. Alternately, they will create alternate streams of revenue and funnel the revenue into pricing.

Access to similar people is important because of the concept of the long tail. The minute I can find a large enough group of people interested in yellow doo-dads, I can afford to make, market, store, pick, pack and ship yellow doo-dads. The internet makes it easier and easier to find and/or create communities of people who are interested in something specific. This means no longer having to stock the products that will appeal only to the widest group of people. E-commerce will enable the carrying and selling of more and more niche products as communities pop up across the country that are willing to support and purchase such products. If you need at least 100 willing customers for yellow doo-dads, but they are spread across the country it will be impossible for a local store to carry them. Alternately, a huge chain of stores will carry one in every store in the hopes of finding the one person in that neighborhood who has a need for yellow doo-dads. However e-tailers with the ability to send the product to any corner of the country can and will move into such opportunities as they arise.

What does this mean for e-commerce? While I can’t claim to know the answer, here is what I think will happen.

E-commerce, or atleast parts of it – will be commoditized. Scale will become all important. The ability to buy something online and have it delivered to your house as a particular time and date will become commonplace. The price will be reasonable and well known.
As this happens, products will start to diverge from the lowest common denominator into ever increasing niche markets.
Discovery of products is going to become ever more important, especially as the number of products available online explodes.
Arbiters of good taste will be rewarded as e-commerce becomes a commodity, and ever increasing number of products show up online.
Good taste isn’t objective – but subjective, so it means whatever you think it means. Large amounts of computing power are going to be spent trying to win the race to become trusted sources of good taste for as many people as possible.

There are several changes coming – curation will become more important, supply and demand will not look like it does now, and e-commerce tomorrow will see major innovation.

Disclaimer : The views above are mine and mine alone and do not represent those of my employer in any way shape or form. I am currently employed at Quidsi Inc, which was recently purchased by Amazon.

If you experience this error while installing the latest version fo Rails 3.0.7 as of this date.

ERROR: While generating documentation for rails-3.0.7
... MESSAGE: exit
... RDOC args: --ri --op /usr/lib/ruby/gems/1.8/doc/rails-3.0.7/ri lib --title rails-3.0.7 Documentation --quiet

It’s because you are likely to be on ruby 1.8.7. Upgrading to 1.9.2 seems to solve this problem.

That being said, it doesn’t seem to hamper Rails in anyway.

Recently, I wanted to send email using Google Apps – which requires TLS for their SMTP server.
Using Rails 2.3.5 and putting the following into my environment.rb solved the problem.<
No plugins required.

Maybe this will help someone.

ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.default_content_type = "text/html"
ActionMailer::Base.smtp_settings = {
:enable_starttls_auto => true,
:address => 'smtp.gmail.com',
:port => 587,
:domain => 'GoogleAppsDomain',
:authentication => :plain,
:user_name => 'username@googleappsdomain',
:password => 'password'
}

The key line above clearly is

:enable_starttls_auto => true,

These days, many companies are attempting to get involved in Social Media outlets (TwitFaceSpace, as one of my customers refers to it).

The issue with getting involved in the “social” space (blogs, wiki’s social networks) is the commitment to keeping it up. Many of us as individuals have started blogs, but realized quickly that the ability to keep it going requires tremendous commitment, with little (or no) discernible ROI for a long time.

However is that true for corporations as well? Is there a direct connection between being engaged in the Social Media space, and your bottom line? Seems intuitive that if you do it right (that’s the catch), and your users are engaging with you in these media, then they will also engage with your products, or at least feel an increased affinity to you and your products, that might not have been there before.

The EngagementDb intends on putting that question to rest. The EngagementDb attempts to create a co-relation between user interaction and engagement in social spaces, and the companies bottom line.

It studied several distinct channels in an attempt to determine the extent of user engagement, and interaction. It then proceeded to score each of the 100 most valuable brands as identified by the 2008 BusinessWeek/Interbrand Best Global Brands ranking.

Top 100 Brands (alright, only some of them!)

The top 10 ENGAGEMENTdb brands with their scores are:

1. Starbucks (127)
2. Dell (123)
3. eBay (115)
4. Google (105)
5. Microsoft (103)
6. Thomson Reuters (101)
7. Nike (100)
8. Amazon (88)
9. SAP (86)
10. Tie – Yahoo!/Intel (85)

The EngagementDb classifies organizations into 4 categories based on how, and where they choose to interact with their users.

• Mavens – brands that have made social media a core part of their go-to-market strategies and are very active in many channels; usually driven by dedicated teams assisted by company-wide awareness and participation.
• Butterflies — brands that recognize the need to be in many channels but have only met with real success in a subset of their activities; these companies are usually spread a bit too thin.
• Selectives – brands that focus on just a few channels and excel in those; these efforts are usually initiated by an internal evangelist.
• Wallflowers — brands present in only a few channels and very lightly in those; these brands are sitting on the sidelines and are wary of the risks. They are still trying to figure out the best next steps and investments in social media.

Now, you want to put it to the test don’t you? Well you can – right away. You can even attempt to rank your brand and see where you fit into that ranking. Head on over to the EngagementDb.

P.S. : As an interesting bit of insight – the EngagementDb – a piece of media about social spaces is currently drawing most of its traffic from Arrington (TechCrunch) and Twitter. I have a thought rattling about in my head about the graph of traffic sources for new content. Will try to write a post regarding that later tonight.

A gentleman whom I’ve learnt a lot from over the last few years, once mentioned to me that he wanted to write a book, called the Business of Measurement. Since he hasn’t written it yet, I will humbly provide this as a potential preview to what the book might be about…

Most sites have analytics so they can track whatever statistics they consider important.

Most times, this falls into 3 categories :

• Page Views
• Visits
• [Unique] Visitors

If you’ve dealt with me ever you are aware of my views on the flaws inherent to the definitions of the above. For those of you lucky enough to have never had to deal with me, here’s a quick definition of each

Google Analytics Dashboard

Page Views : :

Let’s assume that you came to this blog, via a search engine, and landed on this particular post (http://dragonsbehere.com/weblog/e-commerce-platform/analytics-vs-analysis). In my Analytics solution now this page will record an increased Page View count of 1.

If you were to reload this page, that number would go up again.

If you were to find my writing fascinating, and go to other posts, those pages would also incur an increased Page View Count.

Page View has become a slightly muddy issue with the rise of AJAX. What exactly counts as a Page View, when sections of the page change without the page reloading? What percentage of the page has to change before it is a new Page View? What about just counting the elements of the page?

There’s also the issue of Javascript vs Cookies vs Sessions etc. What happens if someone disables Javascript, or third-party cookies? Some of these issues were brought up at a panel a while ago (a long while ago). You can read the answers here.

Visits ::

You’re a visitor.  You opening this page on your computer constitutes a visit to my weblog (for which I thank you). Visits and Page Views usually have a 1 to many relationship – i.e. for every visit, at least one page view is recorded, and possibly more (see above regarding my fascinating writing skills). It is impossible to have more Page Views than Visits.

Obviously Visits suffers from the same problems as Page Views (Ajax, Javascript & cookies disabled) etc. It also has its own set of new issues – what constitutes a visit? What happens, if I leave the page open for session.length + 1 minute? Maybe we could get around that with an infinite session, but then, what is that session based on? Cookies? What if they are disabled, or deleted? How about storing session data on the server? (Note, for the “Shared Nothing” purists, I support your views, just making a point. Also I think it is feasible to do sessions via Db – even in a shared nothing scenario.)

Regardless – as you can see, all approaches have a positive and a negative side to them

[Unique] Visitors::

Visitors, fairly self-explanatory – for example, you’re a visitor to my blog.  Unique Visitors is meant to distinguish between people making multiple visits to the site, and just the number of people visiting the site. Visitors should have a 1 to many relationship with Visits. i.e. for every Visitor, we should record at least one visit. Similarly, Unique Visitors should have a 1 to many relationship with Visitors. You can’t have more Unique Visitors than Visitors. Similarly you can’t have more Visitors than Visits.

Visitors have their own share of issues – what happens if you visit the website from home, and then again from work? What happens if you access it from a public computer (such as a library)? What happens if you can’t store cookie (because you work for the Government) and all visits from your office (lets say – the Pentagon for example) come from a single IP address? Should all those visits be classified as a single visit?

Now, these are just the start of the Analytics part of the discussion. We still have to get to sources of traffic, pathing, landing pages, bounce rates, goal achievements etc. and we will – in future blog posts.

More importantly, what is the point of all these statistics? How does it matter how many people visited your website? Or how many Page Views you generated?

Now, that’s the analysis portion. Sure you can get a metric ton of information on what is happening on your website. Yes, there are plenty of flaws in the gathering of data inherent to the fact that the Internet is a fault tolerant system, and analytics is traditionally not.

What you do with this data is where the secret sauce lies. For example, Page Views could indicate what pages are popular, which in this case (a blog) indicates what content my audience is interested in, and what it isn’t.

Maybe I want to cater to what my audience is interested in, or maybe it indicates that I need to go seek out the audience that would appreciate the rest of my content as well. For example, in my case, the Technology posts get much higher traffic than the E-Commerce posts.

Regardless, analytics without analysis is almost pointless. I say almost because the mere act of measuring something, can cause us to change our behaviors to improve it. Which is the very point of this post. Make sure you are measuring what you want to improve. Make sure your analytical output is closely tied to your business goals.

Let’s take a rather simplistic example :

You run an e-commerce store that experiences the following basic traffic patterns :

• Visitors – 100 /day
• Visits – 200 / day (or two Visits per Visitor)
• Page Views – 400 /day (or two PV per Visit, or four PV per Visitor)

The website experiences a conversion rate of 1% based on Visitors (so 1 out of every 100 Visitors will engage in some action that the business considers worthwhile.)

Now, lets say that the business is measuring Page Views for instance, and decides to increase that number, hoping for it to all trickle downstream – leading to an increased bottom line. Well, there’s a number of ways to do this – let’s consider two of them that I’ve seen used in multiple places :

• Buy Traffic
• Change your UI to require multiple Page Views to achieve the same result.

Both these strategies should increase Page Views. However, the trickle down effect of increasing Page Views for an act of conversion is debatable. Will the bought traffic actually convert at the same rate as the previous (possibly organic) traffic was converting? Maybe, if its bought from a fantastically targeted program. Changing your UI to require more Page Views seems like it would definitely hurt your conversion rate – but in some cases (usually involving eyeballs on ad’s) it might help your bottom line.

As you can tell, there is no silver bullet for the analysis aspect. I can only suggest what my mentor told me. Understand the bottom line. Once it is known what the business measures itself against, whether that’s sales, or profitability, or user happiness, or some combination of those : once that can be succinctly explained, then you know what to measure.

You can’t improve what you can’t measure. Corollary : measuring the ‘wrong thing’, will lead to improving the wrong thing, possibly at the cost of the ‘right thing’.

Over the last few years a number of “personalization” engines have entered the market – and rightfully so. Personalization is the holy grail of merchandising, the magical ability to present the individual, with that product or products that matches his or her needs perfectly.

The problem is that information isn’t distributed equally between the actors in this play. The shopper in question knows a lot more about themselves than the site. The e-commerce site, only knows what it can know, and most times, that is not very much about an individual shopper, even if they force you to login (ala Amazon).

So with this asymmetrical information distribution, the goal is not so much perfection, as it is to best optimize available information.

Amazon claims they have recommendations just for me...

So with this asymmetrical information distribution, the goal is not so much perfection, as it is to best optimize available information.

Before we dive further into the concept of personalization – lets take a quick step back, and talk about ‘merchandising’. Most retailers have a product/product-line that draws in the crowds, but where their margins are very slim. Camera’s for most electronics retailers, for example. These retailers are hoping to sell you accessories, warranties, professional services, whatever they can around the primary product in order to actually realize a profit in the transaction.

Many times, this is encouraged, and achieved via a simple Product X goes well with Product Y type relationship. A good example of this is someone like SamAsh. See the Things You’ll Want section below.

Samash - Things you'll get VS Things you'll want!

This helps the customer surmount the major irritation of waiting excitedly to receive his acoustic-electric guitar only to find that he doesn’t know how to tune it, or that he needs an Amp to sound like Hendrix… stuff that any salesperson in a store would’ve told him right away.

The problem, even before this one however in our story line described above, would have been finding the right product (in this case a guitar, but could as easily have been a camera). That’s where merchandising zones like “People who viewed this, also viewed” (Overstock.com) – or “People who viewed this ultimately bought…” (Amazon.com)

Overstock.com - People who viewed this, also viewed that...

So, thus far, the story goes - a person entered the store, wandered the aisles, answered a few questions the salesperson asked, and voila, found a product. While looking at this product, “we” mentioned a few other products she might be interested in, and in doing so, helped her find the product she really wanted.

Then once we’d found her product, “we” showed her all the other stuff she’d need in order to get the optimal usage out of it.

Twice, in that interaction, we “suggested” products to her. How though, do we know what products to best suggest to someone?  Now - we’re at the crux of what personalization does.

Let’s go back to the store metaphor. Most times, when someone walks into a store, a salesman can ask a few questions, and assuming he knows his stuff (I know, that is indeed, someting increasingly rare these days), he should be able to lead you to a product of your choice. If he’s been around a while and he knows your name then he might also recall what other equipment you have purchased, or own, and suggest accessories accordingly.

Why can’t e-commerce stores do this? If we remove the intangibles that an e-commerce store can never know (what the person looks like, what they sound like, what their body language is saying etc.) we are left with several tangible data points that can be used here – past purchases, intent, location, budget, wish lists, other behavior patterns on the website, other people similar to this person etc.

So the idea of personalization is to optimize what we know about the person based on the tangibles, and then present to them a product that we think is best suited for them.

This is the second problem with personalization. Let’s say it’s an ideal world, and I have a ton of knowledge about the person browsing my site – I know her age, her location, the products she’s bought in the past, the average order of her past orders, what price bucket she usually picks while navigating my site (cheapest, middle, high), which shipping option she chooses (needs it today, tomorrow, or no rush), so on and so forth (and there’s a LOT of stuff in the forth part that we haven’t discussed yet)

So now I know all this – let’s say she picks products that are usually in the middle price range. When suggesting a product to her on a Product Detail Page (like SamAsh’s above) can I take this into account? Do I know for every product that I have whether it is a low, mid or high price range product? If I wanted to calculate this, what other products would I compare it against? Do I compare the chromatic tuner to all other chromatic tuners? Or to all other guitar tuners? Or to all tuners including tuning forks?

Does her gender matter? Do I know if women have picked a certain type of tuner over another, all other factors being the same?

As you can see, personalization has multiple aspects to it – we’ve briefly broached the following

• knowing as much as possible about the customer,
• knowing as much as possible about the products
• knowing as much as possible about how customers interact with the products

Whatever data you have on the three points above, then needs to be mangled into a complex algorithm to try and determine the most personalized suggestion that the store can offer. This is where personalization engines started. They were in depth algorithmic approaches to the problem.

However, they soon noticed that retail stores actually had very little mined data about the third point – customers interacting with and purchasing products. So personalization engines extended themselves to listener’s as well – they now sit there, and record all these interactions. Purchases, clicks, links, emails etc. They are collecting and mining this information based on what data the algorithm needs to make decisions. Based on cookies, and logins they are also gathering information on the first point above – the customer.

For the middle point – data on the products, they rely on the retailer.

So now, we can guess at what the  the ‘secret sauce’ is? It is the ability of personalization engines to mine behavioral, and purchasing patterns for data, and their ability to combine this with the daily feed on products that they receive, run it through an algorithm in order to spit out the possible best fit for an individual.

As Moore’s Law continues its onslaught, and as individuals continue to trade privacy for better services, the rapid rise of personalization will continue. This is great for both customers (who will get better suggestions) and vendors (who will, or so the theory goes, sell more products).  Personalization has a long way to go (insert Semantic Web prophecy here) but it is definitely an area where more and more resources will find their way.

Amazon Merchant Transport Utility – it takes all the worry of SOAP out of your daily life!

Recently a customer wanted to explore the option of using AMTU in order to move files to, and receive reports from Amazon. The idea is simple -

• Run AMTU on a local computer
• Designate a folder
• Voila – receive new reports there and any files you put there get uploaded to Amazon (every 5 to 12 minutes depending on the file)

It did actually work exactly as described above on a Windows machine. However, we want to do some other things with the information we get from Amazon – warehouse, and payment provider integration etc.

I mocked together a “clunky” looking application (as described by my Director of Services) that does the above in Rails – and was running it on an Ubuntu machine over at Slicehost.

Now, to get AMTU running on this machine, (AMTU is all Java, it should ‘theoretically’ run everywhere), and I’m off to the races. I toss on the kettle, and run back to my machine expecting to drink a cup of tea as I watch automation make our customer’s life much easier in a few minutes.

Clunk!

AMTU refuses to install properly on Ubuntu.

In order to make this easier for future folks attempting this.

1) You need Java (OpenJava, Sun Java6, Sun Java5 whatever). You can and should check this exists by typing in

java -version

Assuming this works, on to the next point. If this doesn’t work then do something along the lines of
sudo apt-get install sun-java6-jre

2) You need to be able to echo JAVA_HOME.  Also, don’t make the mistake of pointing it at /usr/bin/java – since the installation is actually asking for the directory wherein Java was installed – it appends the /bin/java part on top of that during the installation. In order to be able to accomplish this in the easiest fashion possible do the following :

sudo nano /etc/environment

Feel free to use any other editor if you like – vi etc. To this file add the line :

JAVA_HOME="/usr/lib/jvm/java-1.5.0-sun"

Note, make sure you actually change that to the file you’re looking for. If you don’t know where you need to point JAVA_HOME, you can get a hint by doing

whereis javac

and then following the symlinks back from that directory to the actual location of javac then traverse the directory tree so that you’re in the directory that has a child named bin, and that bin has a child named java, and you’re in the right place.  In the example below, /usr/lib/jvm/java-1.5.0-sun is the appropriate directory for JAVA_HOME.

Setting JAVA_HOME in Ubuntu (Click on it for a larger view)

3) Now you should be able to just run ./install.sh from inside the AMTU folder.

4) If all went according to plan, this worked great. If it didn’t, you’re probably seeing a -bash XXXXXXXX No such file or folder error.

If that’s the case, then you need to keep in mind two things. One, AMTU’s last release was in 2007. It has no real support in the community, or from Amazon itself. Two, you’re probably on a 64 bit machine, and you’ve never needed to install ia32-libs. You can correct this now by doing

apt-get install ia32-libs

5) Now go ahead, uninstall, and reinstall AMTU.

If you run into any other errors, feel free to ping me, and I’ll share with you what little I know about it.

UPDATE : I got an email claiming that they were unable to find sun-java6-jre as a package. The most likely culprit there is that apt-get isn’t checking the multiverse for packages. If you want more details on what that means – check out the explanation of main, restricted, universe, and multiverse here.

In order to let Ubuntu look into the multiverse for packages that are “not free” you need to change the following files /etc/apt/sources.list

sudo nano /etc/apt/sources.list

Your file probably looks similar to this :

deb http://archive.ubuntu.com/ubuntu/ hardy main restricted universe
deb-src http://archive.ubuntu.com/ubuntu/ hardy main restricted universe
deb http://archive.ubuntu.com/ubuntu/ hardy-updates main restricted universe
deb-src http://archive.ubuntu.com/ubuntu/ hardy-updates main restricted universe
deb http://security.ubuntu.com/ubuntu hardy-security main restricted universe
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted universe

You want your file to look like this :

deb http://archive.ubuntu.com/ubuntu/ hardy main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ hardy main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ hardy-updates main restricted universe
deb-src http://archive.ubuntu.com/ubuntu/ hardy-updates main restricted universe
deb http://security.ubuntu.com/ubuntu hardy-security main restricted universe
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted universe

I know it doesn’t seem that way when you visit most stores today, (especially as you spend hours looking for an electric razor that is inexplicably no longer in the “bath” section, but rather is in the “electronics” section due to its battery powered nature; sorry I digress) but the layout and placement of items in a store is a carefully controlled science that many corporations like Coca Cola and Walmart have invested tons of money into. The reason is simple – they want to maximize what limited shelf space they have available. What they put on the shelves, where they put them, how you see them, etc. all has an effect on how you spend your money.

Shelf Space

There’s a widely held misconception that this shelf space/real estate limitation is no longer applicable online. Since stores are not limited by the “square footage” available to them, they feel freed from the constraints that are enforced on brick and mortar stores.

However this isn’t always true. The metaphors that e-commerce was born with, still apply – aisles become categories, and properties such as brands, price etc. that are used to determine how products are placed on shelves become refinement options.

There’s still a limit on real estate, but this time its enforced by the attention span of the visitors, and the bounce rate experienced on every click.

So how does this ‘limitation’ manifest itself.

Consider the ‘web page’ as seen in a browser window. Depending on the resolution of the viewers screen – you could have anywhere from 800 x 600 to 1200 x 900 and more real estate to work with. Everything else – whether its a click or a scroll away, is area that might not be seen by the visitor, either because they chose not to make that extra click, or they chose not to scroll down… Keep in mind that not all ‘folds’ are created equal – and some might even argue that the concept of above and below the fold is quickly becoming obsolete.

Still, for now let’s make the argument that every action required by the user has a certain bounce rate.

So – how does one maximize the use of their real estate?

Data Trumps Intuition as seen in this slide from the Amazon Deck

In the last post, I discussed some of the challenges around creation, acquisition and management of content. This post I want to look at the problems with distribution.

Content syndication has been around for a while for most types of publishers. There’s a lot of ways to consume information, away from its source of creation. These include RSS readers, aggregators, etc.

An example of feed flare for publishers

However, this hasn’t really been the case for e-commerce. Content syndication has thus far been limited to Comparison shopping engines, and in some select types of content (such as reviews) the content is aggregated together by a third party such as Buzzillions.

Why is this?

Shopping (especially in certain verticals such as fashion, home, and for almost all purchases above a certain ticket price)  is a social activity (whether you’re consulting your friends, your wife, your bff etc.) Yet this social component of the shopping experience has been slow to move over to the web. For the most part e-commerce is a one to one experience – and its tough to incorporate a social component into it without calling your consultant over to the screen to look at the product you’re looking at. This makes, “distributed” shopping very difficult.

Only recently have things like “share on facebook” started to pop up on sites such as Dr.Jays (fashion) etc.

Share on Facebook on sites such as Dr.Jays.com

This content syndication, and sharing is still in its infancy. Much more is possible, as should be obvious by taking a quick look at the publisher’s toolset.

As companies like BestBuy start to create open API’s to their data and structure and format of e-commerce data starts to follow a standard, we will see the democratization of content. Take a look at what Bestbuy offers below :

Best Buy offers tons of product related data

As you can see – it includes data for historical, and current products including near real time availability information for both online, and retail store locations. This is merely a first step.  Imagine the types of applications that could be built using data like this from all vendors? People “rolling” their own comparison engines that are limited to a few stores, and a few verticals that they are interested in. People getting RSS feeds of all products reviewed by at least 50 people that have a rating over 4 stars across all vendors, but limited to products that are in the category “Vespa Accessories”. All these slices, and dices should be possible today, but aren’t because of this limited access to data.

Sites like Fixya or RateItAll are trying to build this meta social layer, but they struggle with the basic information on each product – descriptions, specifications, features, images, manuals, etc. from the manufacturers and availability, pricing, shipping, and other value added content from the retailers.

Fixya.com attempts to build a social support layer for consumers

Once we have access to complete, well formatted, standardized data – what would be the effort involved in taking the next step and building the ability to socially share information about it. Imagine an application that makes shopping as social as PhotoPhlow makes photograph discussion. If you haven’t seen Photophlow yet – watch the video below.

This ability to discuss, share, talk in real time, or asynchronously will lead to a radical change in the way people interact with e-commerce sites. As data standards start to form, more and more content will be user generated. We can already see some of this in the form of user submitted pictures on Amazon, and certainly in reviews and ratings.

Images with notes as submitted by users on amazon.com

This data format standardization and availability will enable the widespread syndication, and sharing of data, and the merging of the manufacturer, and retailers supplied data with user generated data (the beginnings of which can be seen above). This in turn will enable a wide set of applications developed to help people shop in a wider variety of methods. At this intersection of widespread disemmanation of data, and social interaction between users, lies a gold mine of analytics data that will help surface, what people are buying, what the trends are, what the gating factors are – as well as who the mavens, and connectors are across the entire industry. I for one am excited to see it coming.

What else do you think we could build if we had easy access to clean e-commerce data?

Content has traditionally been a struggle for e-commerce vendors. Brick & mortar stores going online had limited content to provide this new channel, and what data was available was not nearly sufficient to replace the tactile connection shoppers had to products in a store.

Some vendors, such as B&H invested heavily in this area and it proved to be a strong differentiator for them.

Basic E-Commerce content (which I describe as features, specs, description, and images) for areas such as electronics has become a commodity – with companies like CNET, Etlize able to provide this data in a structured format, cheaply and easily. Others like Sellpoint provide more in-depth, multimedia content for products, raising the bar even higher.

So – does this remove the content based differentiators for companies?

Let’s evaluate what this really means…

Companies able to produce detailed content did it because they had domain experts working for them. People who knew what they were talking about – and knew the products inside and out. They were authentic.

Authenticity is something that retailers, bloggers, content producers, and just about everybody should, and does strive for. Authenticity leads to trust, which in this day and age, is a very valuable commodity.

These companies haven’t lost these people – its just become harder for them to prove their authenticity. There is a sea of vendors that provide similar data online. How does one differentiate themselves?

As mentioned earlier – companies like etilize, sellpoint, cnet etc. have raised the bar on what data can be purchased by vendors. Companies like B&H now need more than just the basic data to be able to utilize it as a differentiator. They now need to provide more and, better data, in more formats to allow customers to consume it any way they desire, whenever, and wherever they want.

The challenge here isn’t just limited to the management of the data created by the in house team. It extends to the correct classification, structure, and storage of data that vendors are gathering from third parties – like the manufactureres themselves, companies like cnet, etilize and sellpoint, as well as User Generated Content like reviews or ratings.  It also means that they have to be able to provide this data for any output format desired – whether they are catalogs, brochures, the website, a mobile campaign, etc.

The internal content creation itself can present a challenge. Ideally, all the information that rests with each individual Domain Expert should be frictionlessly shared among all of them. The customer experience should be the same no matter what channel they choose to interact over. This requires that all the data should be available in all the channels.

Getting everybody to share their data in a coherent, structured format – is more of a challenge than it sounds. Even little things like formatting, spellings etc might trip up systems around guided navigation etc. (example a search engine won’t see Black, and blk as the same thing, when creating navigation options ). Creating the templates, with the appropriate validation, error checking, formatting, and structure for your domain experts to use is another piece of the challenge.

As you can see – “data” can be a hairy issue. Especially product data that needs to be uniform, clean and structured, for purposes such as search, browse, print, catalogs, ads, and marketing initiatives.

There are some solutions to the problem. Technologies such as Produt Information Management systems are reaching a level of maturity that allows for unprecedented control and flexibility over content.  (PIM’s are not the same as a CMS, but that’s a post for another day).

There are many different approaches to the solution – every company (Fulltilt, SteboSystems, Hybris etc.) have approached it differently, but the end goal is very similar. Clean, managemeable, structured, data that can be imported from, and exported to,  almost any location in any format, and style.

If any of this resonates with you, or if you have any experience in this – please let me know if you think I’ve missed something, or if there’s anything that should be added to the challenges faced during the creation and maintenance of content.

In the next post I’ll briefly discuss what I think can be achieved if retailers and manufacturers had the ability to create, maintain, and distribute data, in a clean, standardized format.