A database shard, or simply a shard, is a horizontal partition of data within a database or search engine. Each shard may be held on a separate database server instance in order to spread across multiple servers.

Some data in a database may remain present in all shards, while other data is stored in only one shard. In such cases, each shard acts as the single source for its subset of data.

Horizontal partitioning is a database design principle whereby rows of a database table are held separately, rather than being split into columns (as in normalization and vertical partitioning, to varying degrees). Each partition forms part of a shard, which may in turn be located on a separate database server or in a separate physical location.

There are numerous advantages to the horizontal partitioning of data. Since tables are divided and distributed into multiple servers, the total number of rows in each table in each database is reduced. This reduces index size, which generally improves search performance. A database shard can be placed on separate hardware, and multiple shards can be placed on multiple machines. This enables the distribution of a database across a large number of machines, which can significantly improve performance.

In addition, if the database shard is based on some real-world segmentation of the data (e.g., European customers v. American customers) it may be possible to infer the appropriate shard membership easily and automatically, and to query only the relevant shard.

In practice, sharding is complex. Although it has long been implemented through manual coding (especially where rows have an obvious grouping, as in the customer region example above), this approach is often inflexible. There is a desire to support sharding automatically, both in terms of adding code support for it, and for identifying candidates to be sharded separately. Consistent hashing is a technique used in sharding to distribute large loads across multiple smaller services and servers.

Horizontal partitioning splits one or more tables by row, usually within a single instance of a schema and a database server. It may offer an advantage by reducing index size (and thus search effort), provided there is an obvious, robust, and implicit way to identify in which partition a particular row will be found, without having to first search the index; for example, the classic case of the 'CustomersEast' and 'CustomersWest' tables, where a ZIP code already indicates where a row will be found.

Sharding extends this approach. It partitions the relevant table or tables in the same way, but does so across potentially multiple instances of the schema. An advantage is that the search load for the large partitioned table can be distributed across multiple servers (logical or physical), rather than only across multiple indexes on a same logical server.

Splitting shards across multiple isolated instances requires more than simple horizontal partitioning. The expected gains in efficiency would be reduced if querying the database required multiple instances to be accessed just to retrieve a simple dimension table. Beyond partitioning, sharding therefore involves distributing large, partitionable tables across servers, while smaller tables are replicated in full on each server.

Sharding a database table before it has been optimized locally can introduce unnecessary complexity. Sharding is generally recommended when other optimization strategies have proven insufficient. The added complexity of database sharding can result in several potential challenges.

In a database context, the term "shard" has two independent points of origin. One is Computer Corporation of America's "SHARD" (System for Highly Available Replicated Data), a distributed database architecture in which every node holds a full replica of the database rather than a distinct partition of it. Despite the name, this system is unrelated to horizontal partitioning as the term is used today.

The other is the 1997 MMORPG Ultima Online. Richard Garriott, creator of Ultima Online, recalled that the term originated during the production of the game, specifically in creating a self-regulating, virtual ecology system. Players were able to interact and harvest in-game resources, which disrupted the balance of the system.

To address this, the development team divided the player base across multiple parallel copies of the game world, and introduced a fictional justification tied to the end of Ultima I: The First Age of Darkness, where the defeat of its antagonist Mondain also led to the creation of multiverse "shards." This gave Garriott's team the in-universe rationale for running multiple copies of the environment. It remains a core part of Ultima Online's architecture, with dozens of shards still active.