Oyster Storage Now Supports 25MB Uploads!

The  Oyster development team is pleased to announce that the initial file  size limit of 5MB associated with the mainnet launch of the Oyster  Storage platform has been increased to 25MB.The  development team optimized several parts of the platform to increase  the original file size limit from 5MB to 25MB. For example, previously  the client would process the entire file before sending all the chunks.  For extensive files, this would cause memory crashes in the browser  because the client had to be holding the entire processed file. Now,  Oyster Storage handles the file in batches, and sends the chunks in  batches, avoiding the client-side out-of-memory crashes. Additionally,  all the data for a chunk was stored in a MySQL row, including the  “message” field which is 2187 characters long and was burdensome on  Oyster’s queries. Now, the development team has moved this large field  to Badger, which is a key-value database. The key for the message field  is still stored in the MySQL row, but it is considerably smaller than  the message field. The team plans to move further and further away from  MySQL for data map chunks to further improve optimization. Another  change made was that chunk data is now stored in the Badger database and  other data about a chunk (status, address, etc.) is stored in MySQL.  Previously, these two databases were somewhat coupled; the badger data  could not be added to the broker node until the MySQL data was added.  Now, these two processes are independent of one another, which means the  client can go ahead and start sending chunk data as soon as the session  has been created, instead of waiting for the broker node to build the  data map (which can take a long time for large files).We  ask our community to note that the team is aware of concurrency  limitations. If many users upload large files simultaneously, uploads  will slow down considerably. We are working on addressing this issue.In  addition to this file size increase, the development team already has  some ideas to push the file size limit to an even larger limit.  Currently, Oyster Storage is capable of handling file uploads of ~100MB;  however, there is a concurrency issue in the infrastructure, such that  if many individuals try to upload files of this magnitude at the same  time, it is likely that the upload will timeout before it is finished.  Once the development team has implemented their ideas to deal with these  concurrency issues, they will begin further testing of larger file  uploads (~100Mb) to ensure that many users can upload files of this  magnitude concurrently without a timeout occurring.Some of the changes that the development team are exploring to reach larger upload sizes and higher overall throughput are:

• implement round-robinning of uploads so that every user’s upload receives a little attention
• continue to depend less on MySQL
• switch to Hercules when it is released
• use AWS Lambdas (a form of serverless functionality) to offload the Proof-of-Work burden elsewhere
• add more broker nodes to Oyster’s network

The  development team will be working tirelessly to improve the file size  limit even further, and we will continually update our community  regarding the progress made towards this goal.Try out the new file size limit here: https://oysterstorage.com/