Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DSIP-][api-server] dispath the resource to worker-server #15938

Closed
2 tasks done
wangxj3 opened this issue Apr 29, 2024 · 3 comments
Closed
2 tasks done

[DSIP-][api-server] dispath the resource to worker-server #15938

wangxj3 opened this issue Apr 29, 2024 · 3 comments
Labels
wontfix This will not be worked on

Comments

@wangxj3
Copy link
Contributor

wangxj3 commented Apr 29, 2024

Search before asking

  • I had searched in the DSIP and found no similar DSIP.

Motivation

The file upload by the page of resource ,the file will save by the api-server on server,if there is no worker-server in this server ,and the DS group has no share file system ,it's hard for user to use this file .
通过资源页面上传文件,该文件将通过api服务保存到api所在的服务器,如果该服务器中没有worker-server,并且DS组没有共享文件系统,用户将很难使用该文件。
对于简单实用的用户,应该间可能减少部署成本。
对于核心使用场景,要保证系统的稳定性(包括依赖插件稳定性原因导致的DS集群稳定性)。

Design Detail

No response

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

No response

Code of Conduct

@wangxj3 wangxj3 added Waiting for reply Waiting for reply DSIP labels Apr 29, 2024
Copy link

Search before asking

  • I had searched in the DSIP and found no similar DSIP.

Motivation

The file upload by the page of resource ,the file will save by the api-server on server ,if there is no worker-server in this server ,and the DS group has no share file system ,it's hard for user to use this file .
Upload a file through the resource page, and the file will be saved to the server where the api is located through the api service. If there is no worker-server in the server, and the DS group does not have a shared file system, it will be difficult for users to use the file.
For simple and practical users, it should be possible to reduce deployment costs.
For core usage scenarios, it is necessary to ensure the stability of the system (including the stability of the DS cluster caused by dependence on plug-in stability).

Design Detail

No response

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

No response

Code of Conduct

@ruanwenjun
Copy link
Member

ruanwenjun commented Apr 29, 2024

I am -1 to this DSIP.

In standalone mode, users can directly use the local file system. In cluster mode, users can use distribution file system or some shared file system like NFS, the distribution file system is very commonly, this is not an expensive technology.

Back to the implementation, do you understand the complexity of implementing a file system? The only thing I can think of as an implemenration is we broadcast the file to all workers, but this means the cluster will be hard to scaled.

In additional, DS is not must rely on a distribution file system, this is different with spark/flink which is stronge rely on a distribution file system to store the checkpoint data, these systems still don't have plan to implement a file system.

@ruanwenjun ruanwenjun removed the Waiting for reply Waiting for reply label Apr 29, 2024
@caishunfeng
Copy link
Contributor

In cluster mode, users can use distribution file system or some shared file system like NFS, the distribution file system is very commonly, this is not an expensive technology.

+1, DS should pay more attention to its own schedule business, and do not need to add more logic of the basic file system.
This will introduce more unnecessary complexity and maintenance costs.

@SbloodyS SbloodyS added wontfix This will not be worked on and removed DSIP labels May 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

4 participants