TMall 11.11: millisecond-level update on search engine

Posted by CodingCat on November 13, 2014

In China, Nov 11th is called “Single Day”, the original meaning of which is the celebration day of the people with single status.  Nowadays, this day becomes the largest shopping festival in the world (imagine the population in China…:-) ) TMall is the B2C service of Alibaba group. On Nov 11th, 2014, the total transaction value of TMall is round 9.9 billion dollars. Put the business value aside, since the first Single Day, the technical challenge faced  by TMall is the #1 topic in the tech-community in China.

In this article, I will introduce the latest search engine architecture of TMall. 

Search engine is usually the first entry of the TMall users. Users input keywords and gets the search result which is sorted based on total sales, inventory, popularity, etc. and the order of the results usually has the high impact to the users’ final decision. Under the high traffic pressure on “11.11”, how to show the result accurately and in real time becomes one of the largest challenges faced by the technical team in TMall.


Tmall search engine

The above graph shows the architecture of the TMall search engine. The application layer is the interface for the client to access the service, and the business layer adjust the presentation result according to the business requirement (e.g. Android and iOS have the different results. The search engine layer is more like the traditional search engine, but TMall also introduces the adjustment to the search results based on the user behavior. The indexing layer contains full indexing (offline) and streaming computing (online). In this layer, to avoid making the infrastructure more complicate, they use YARN and HBase for computation and storage  instead introducing additional Storm. The DB layer synchronizes the product, user and transaction info to the HBase in the indexing layer.

Streaming Computing

The other for TMall to discard Storm is that, Storm is a stateless computing framework, which fits more on the simple scenario, e.g. some simple statistical counting. Alibaba team has the self-made iStream framework, which is with state and utilize HBase to store user status to complete more complicate computation. At the same time they push the result to the upper layer (search engine layer).

Sorting Chain

In the search engine, the personalization service will act based on the real time feedback from the user, and Query Planner will analyze the keywords, scenario, current page and provide further personalized service.

In the business layer, TMall implements the sorting chain to sort the search result based on the business scenario. The sorting algorithms are organized as the plugins and can be combined for different scenarios. TMall system will apply the sorting chain to the search result based on context the user is currently in (business requirement, user type, scenario, query itself, etc.)

Optimization for 11. 11

In the scenario of 11.11, one of the biggest challenges faced by the TMall search engine is that, the result should not contain the “sold out” products, while some products which are very very popular can be sold out within several minutes. To provide the “real-time” result, TMall search engine removed the cache component in the business layer (since they cannot be aware of the external change, e.g. the change of the inventory ), but put more effort to optimize the cahce in the search engine layer. Taken the “filtering the sold-out products” as the example, the external system sends an asynchronous message to the indexing computing cluster, which updates the index. When the search engine layer returns the result, it will filter the cached result to ensure the accuracy of the results.