找到你要的答案

Q:When specifying local[n1,n2,n3] for spark master, what are the three parameters?

Q:在指定的地方[ N1、N2、N3 ]火花的主人,三个参数是什么?

For launching spark, I have seen:

--master local[n1,n2,n3] 

where n1, n2, and n3 are integers.

What do these refer to ?

为了发射火花,我已经看到:

--master local[n1,n2,n3] 

在N1、N2和N3是整数。

这些是指什么?

answer1: 回答1:

The master specification is parsed in SparkContext.createTaskScheduler. (See the link for the implementation.) The possibilities with local are:

  • local uses 1 thread.
  • local[N] uses N threads.
  • local[*] uses as many threads as there are cores.
  • local[N, M] and local[*, M] are like above, but set the maximal task failures to M. This allows you to enable retries when running locally. (Normally local retries are disabled. Enabling them is useful for testing.)
  • local-cluster[numSlaves, coresPerSlave, memoryPerSlave] starts executors in separate processes as configured, but it does not require running workers and masters. It's a lightweight way to simulate a cluster in unit tests. (See also SPARK-595.)

掌握规范sparkcontext.createtaskscheduler解析。(见链接的实现)的可能性与当地:

  • local uses 1 thread.
  • local[N] uses N threads.
  • local[*] uses as many threads as there are cores.
  • local[N, M] and local[*, M] are like above, but set the maximal task failures to M. This allows you to enable retries when running locally. (Normally local retries are disabled. Enabling them is useful for testing.)
  • local-cluster[numSlaves, coresPerSlave, memoryPerSlave] starts executors in separate processes as configured, but it does not require running workers and masters. It's a lightweight way to simulate a cluster in unit tests. (See also SPARK-595.)
answer2: 回答2:

only these parameters are supported for "local" master mode :

  • local : one thread
  • local[n] : n threads
  • local[*] : as much thread as possible considering your CPUs

c.f. https://spark.apache.org/docs/latest/submitting-applications.html#master-urls

Regards,

Olivier.

只有这些参数支持“本地”主模式:

  • local : one thread
  • local[n] : n threads
  • local[*] : as much thread as possible considering your CPUs

例如HTTPS:/ /火花。Apache。org /文档/新/递交申请。#掌握HTML的URL

当做,

奥利维尔。

apache-spark