找到你要的答案

Q:About choose buffer size in HttpWebRequest GetResponseStream

Q:关于选择HttpWebRequest GetResponseStream缓冲区大小

var h = (HttpWebRequest)WebRequest.Create(url);
using (var hr = (HttpWebResponse)(await h.GetResponseAsync()))
{
    using (var s = hr.GetResponseStream())
    {
        using (var f = new FileStream(saveTo, FileMode.Create, FileAccess.Write, FileShare.None))
        {
            int bytesCount = 0;
            byte[] buf = new byte[2048]; //<------------------------------

            while ((bytesCount = await s.ReadAsync(buf, 0, buf.Length)) > 0)
            {
                await f.WriteAsync(buf, 0, bytesSize);
                // Update UI : downloaded size, percent,...
            }
        }
    }
}

I'm writing a downloader support update UI (ObservableCollection of thounsands items - Batch download) when download progress changed and resume download but not support multi-segment download (as each item's size ussually < 10MB).
I run about 5-20 downloads concurrency. What buffer size is suitable for this case (good for both UI update and for the download)?

var h = (HttpWebRequest)WebRequest.Create(url);
using (var hr = (HttpWebResponse)(await h.GetResponseAsync()))
{
    using (var s = hr.GetResponseStream())
    {
        using (var f = new FileStream(saveTo, FileMode.Create, FileAccess.Write, FileShare.None))
        {
            int bytesCount = 0;
            byte[] buf = new byte[2048]; //<------------------------------

            while ((bytesCount = await s.ReadAsync(buf, 0, buf.Length)) > 0)
            {
                await f.WriteAsync(buf, 0, bytesSize);
                // Update UI : downloaded size, percent,...
            }
        }
    }
}

I'm writing a downloader support update UI (ObservableCollection of thounsands items - Batch download) when download progress changed and resume download but not support multi-segment download (as each item's size ussually < 10MB).
I run about 5-20 downloads concurrency. What buffer size is suitable for this case (good for both UI update and for the download)?

answer1: 回答1:

You want to use a buffer size that is a multiple of the OS page size, because that is the granularity for writes to disk and pages in memory. Using anything smaller than an OS page size will be suboptimal.

OS pages are generally 4096 bytes. The default buffer size for a FileStream, used if no buffer size is provided during its construction is also 4096 bytes.

For disk I/O it is generally preferable to have a buffer that is somewhat larger (32-128 KB).

In your scenario, using a maximum of 20 concurrent downloads, if you were to use a buffer size of 32 or 64 KB, this would only require 640 KB or 1.2 MB of memory, so those are clearly viable options.

Let's assume you are in the USA, where the average download speeds are 23 Mbps and 12 Mbps for broadband and mobile respectively, then if you were using 64 KB buffers (1.2 MB for 20 concurrent downloads), you could update the UI for each of the 20 downloads several times per second.

So, use at least 32 - 64 KB buffers.

One thing to take care off, is not to continuously allocate new byte buffers, but recycle these fixed size buffers by using a buffer pool

您要使用一个缓冲区大小,这是OS页面大小的倍数,因为这是在内存中写入磁盘和页面的粒度。使用小于OS页面大小的任何东西都将是次优。

OS页面一般为4096字节。对于一个FileStream默认缓冲区大小,如果没有缓冲区的大小是在其建设提供用也是4096字节。

磁盘I/O通常最好有一个稍大的缓冲区(32到128个KB)。

在这样的情况下,使用20个并行下载,如果你使用了一个32或64 KB的缓冲区的大小,则只需要640 KB或1.2 MB的内存,所以它们显然是可行的选择。

让我们假设你是在美国,那里的平均下载速度是23 Mbps宽带和移动分别为12 Mbps,那么如果你使用64 KB缓存(1.2 MB 20并发下载),你可以更新每20下载几次每秒的UI。

因此,使用至少32 - 64 KB缓存。

要注意的一点是,不要连续分配新的字节缓冲区,而是使用缓冲池回收这些固定大小的缓冲区。

c#  httpwebrequest  size  buffer