Let’s design a file hosting and sharing services like Dropbox or Drive. Today, cloud file storage services have become very popular as they simplify the storage and sharing of digital resources among multiple devices. Access resources from different devices regardless of geographical location is very convenient.
Requirements
Functionality
First, let’s take a look at the design goal of cloud sharing services. A typical cloud drive usually has the following essential functions:
- Users can upload and download their digital resources (files, photos) from any device.
- Users can share digital resources with others.
- Automatical synchronization of resources among devices
- Support storage of large files
- ACID should be a guarantee for all resources.
More advance, we also want to have these functions:
- Users should be able to see all versions of the resources.
- Support off-line editing, and once they come online, the sync should automatically happen among all the devices.
QPS Consideration
The system expects to have huge read and write operations. And the read and write times are nearly the same.
Assume we have 500M users and 100M daily active users (DAU). Each user has an average of 3 devices and storage of 200 files/photos, with an estimation of 100KB per file. Assume each DAU user performs 100 reads and writes per day.
Storage capacity = 100 KB * 200 * 500M = 10 PB
QPS = 100M * 100 / 86400 = 115K/sec
We can consider splitting the file into chunks for dealing with large data files, which can help in both storage, upload, and download. Then we also need metadata for keeping track of chunks. Also, we can consider an incremental or delta upload/download with chunk. If part of the chunk is modified, we can upload the delta of that edited version.
High-Level Design
The core flow will be:
The user can specify a directory as a workspace to create/edit/delete resources. So we need a client service that can interact with users on reads and writes. Also, this client service can handle upload and download files.
At the backend, we have to storage both the files and metadata. We need a metadata manager to handle some info like id, file name, file size, sharing info, etc. And the storage of the files can be chunk storage like we discussed. Then the metadata also needs to store the chunk storage information. E.g.: file id = 1 is split into 10 chunks, with chucnk_id from 0 to 9, the metadata should contain chunk_id = 0, storage_dir = xxx. As for chunk storage, we can have a block storage service responsible for file split based on the files' size and storage to the file storage system in the cloud.
We also need to detect each edit on the files and sync among devices, so we need a synchronized service. Whenever there is an update on the files, this service will sync among services and notify all client services to serve the latest data to users.
Component Design
Let’s go through the detailed design for each component.
Client-Server
Client-server is responsible for view, upload, download, notify for the latest updated resources. This component serves as a control plane of the application. To fast respond and save queries to our backend DB, we can store a copy of metadata in client servers for the view function. For simple browsing operations, this copy of the metadata is good enough for giving information. And per any write operations, the metadata manager will notify the client-server to update the local copy.
Chunk file
To fast transfer files, we can chunk a single file into blocks. Before storing the file in cloud storage, the block server can help calculate the chunk size. We can calculate the chunk size by considering 1) Storage devices in cloud storage for optimized usage, 2) Network transfer bandwidth, 3) average chunk size. Keeping a similar chunk size for each file can save some customized effort. Of course, the metadata of chunk info will be store in the metadata.
Synchronization service
Each client-server needs to listen to changes made for the workspace, and synchronization service can be the one process file update made by clients and applied the changes to all other subscribed devices. There are two ways to do that: PULL and PUSH. Unlike the real-time mobile app, the sync for a drive can happen on-demand. So each client can pull from synchronization service the updates when they are back online. And when there is an update on the metadata, the synchronization service can broadcast to the subscribed clients.
The updates that synchronization service broadcaster should be delta changes, which can save bandwidths. To meet this design goal, we can have a different algorithm to find the difference between the two versions of the resources. Further, as chunks transfer the files, we can calculate each chunk’s difference and send fewer data. We can do to hash each chunk (SHA-256), then we compare the hash between the versions, and send the chunk only if the hash is different, which means an update.
Metadata
The metadata is responsible for maintaining the versioning and metadata information of the files/resources. Consider that our system requires ACID for data consistency, and it is not a real-time app, so latency doesn’t have a strict limit. So SQL DB will be a simple solution for our use case. And the schema can be:
Scale
With growing customers, our system should be extensible and scalable.
First, let’s see how to scale the metadata. Partitioning in DB storage is a very common solution for dealing with large datasets. We can partition the data vertically and horizontally. Partition vertically means we can partition our DB so that we store related resources on one server. And as we are getting more data, we can add more hardware and scale it. Another option is we split the data based on the file name. Let’s say files started with ‘a’ to ‘e’ will store on server A. Next files started with ‘f’ to ‘h’ will be stored on server B. The cons of this approach are easy leads to an unbalance of data storage among servers. A better way for partitioning data is we can do Hash-Based partitioning. In this scheme, we take a hash of the storage object and hash it to a key. Based on the hash key, we figure out which DB to choose. Further, we can also use a consistent hash for solving the unbalance problem.
Caching can be used to increase response speed to customers. We can store some hot resources like recent files from the block server in the cache. We can learn from Memcached that can store the whole chunk with its representative ID/Hashes and Block servers before hitting the block server to quickly check if the cache has the desired data.
