找到你要的答案

Q:Download large file on Google App Engine Python

Q:下载谷歌App Engine Python大文件

On my appspot website, I use a third party API to query a large amount of data. The user then downloads the data in CSV. I know how to generate a csv and download it. The problem is that because the file is huge, I get the DeadlineExceededError.

I have tried tried increasing the fetch deadline to 60 (urlfetch.set_default_fetch_deadline(60)). It doesn't seem reasonable to increase it any further.

What is the appropriate way to tackle this problem on Google App Engine? Is this something where I have to use Task Queue?

Thanks.

在我的appspot网站,我使用三分之一方API查询大量数据。然后用户下载的数据以CSV。我知道如何生成一个CSV下载吧。问题是,因为文件是巨大的,我得到的deadlineexceedederror。

我曾试着增加取截止到60(urlfetch。set_default_fetch_deadline(60))。它似乎不合理增加任何进一步。

什么是适当的方式来解决这个问题在谷歌应用程序引擎?这是我必须使用任务队列的地方吗?

谢谢.

answer1: 回答1:

DeadlineExceededError means that your incoming request took longer than 60 secs, not your UrlFetch call.

Deploy the code to generate the CSV file into a different module that you setup with basic or manual scaling. The URL to download your CSV will become http://module.domain.com

Requests can run indefinitely on modules with basic or manual scaling.

deadlineexceedederror意味着你的请求所花的时间比60秒,不是你的urlfetch呼叫。

部署代码分成不同的模块,你建立基本的或手动缩放生成CSV文件。URL下载您的CSV将成为http://module.domain.com

请求可以无限期地运行在具有基本或手动缩放的模块上。

answer2: 回答2:

Alternately, consider creating a file dynamically in Google Cloud Storage (GCS) with your CSV content. At that point, the file resides in GCS and you have the ability to generate a URL from which they can download the file directly. There are also other options for different auth methods.

You can see documentation on doing this at

https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/

and

https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/functions

Important note: do not use the Files API (which was a common way of dynamically create files in blobstore/gcs) as it has been depracated. Use the above referenced Google Cloud Storage Client API instead.

Of course, you can delete the generated files after they've been successfully downloaded and/or you could run a cron job to expire links/files after a certain time period.

Depending on your specific use case, this might be a more effective path.

另外,考虑到谷歌云存储中动态创建一个文件(GCS)与您的CSV文件的内容。在这一点上,该文件驻留在GCS和你有能力产生一个网址,可以直接下载文件。也有不同的验证方法选择。

你可以看到在做这方面的文档

https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/

https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/functions

重要提示:不要使用文件API(这是一个常见的方式动态创建blobstore / GCS文件)因为它已经depracated。使用上面提到的谷歌云存储客户端API代替。

Of course, you can delete the generated files after they've been successfully downloaded 和/or you could run a cron job to expire links/files after a certain time period.

根据您的具体用例,这可能是一个更有效的路径。

file  google-app-engine  csv  download  google-app-engine-python