- This is a web backend project based on the
flask-restful, coding inpython3.8. - You can treat it as a template to learn web-framework or just simply a start to use python writing web project.
- This project uses Typing for typing hint.
- This project uses Unittest for testing.
- 1. Require
- 2. Run
- 3. REST
- 4. Benefits of Rest
- 5. Unified Response Structure
- 6. Unified Exception Handling
- 7. Unified Query Model, Response Model and DataBaseModel
- 8. Swagger or Not Swagger
- 9. Structure of This Project
- 10. N + 1 Problem
- 11. The Downside of Rest
- 12. Improvements
- 13. Some Tools Should be Mentioned
- 14. A Word
- Maintainers
- Contributing
- License
- Python3.8
- MySql（5.7 or above）
- You can run this project either using Docker or Shell.
- install docker.
- cd to project directory, run
make rununtil it is finished.
make runwill build docker image, start server (Mysql for example).
- run
make initto initial database (create database, create table , No data is imported).
- go to localhost to check if it starts correctly.
Note: make test will run all unit tests in folder tests.
- use commands like
CREATE DATABASE flask DEFAULT CHARSET utf8or GUI tool (such as DataGrip) to create a database in your own DB server. - modify
SQLALCHEMY_DATABASE_URIandSQLALCHEMY_DATABASE_BASEin filesettings.pyto your own setting. - run:
python3.11 -m venv --clear venvsource ./venv/bin/activepip install -r requirements.txtpython server.py- go to the localhost to check if it starts correctly.
- If you want to load soma data to play with this project, use
data.sqlprovided in this project.
- Open DataGrip (a GUI tool for managing with database), connect to Mysql Database;
- Use any editor to open
data.sql, copy it's content to console, execute it;
- Using DataGrip, you should get something like this:
- what is REST ? REST stands for (Resource) Representational State Transfer, it's a stateless communications protocol. The core concept of rest is Resource. In REST point of view, each concept that can be abstracted is called a Resource. Let's say properties
name,age,emailcan be abstract as a User Model, soUsercan be represented as a Resource. - Transfer means resources are transferred from the server-side to the client-side.
- In REST world, each operation is operated on Some kind of resource, and has pre-defined Verb to describe it. Such as Post means to create a resource, Put means to update a resource, Delete means to delete a resource. These three Verb is mainly used, you can check it out here for more detail.
- Using REST, you will get some advantages:
- Each URI (or URL) is for one specific Resource, it makes your code clean and self-describing. Basically, self-describing means what you see is what you get, you can make a quick guess about what you will get from the URI.
- Resource is separated from view function, which means you can change your backend logic function without affecting others, only if you take the same arguments and return the same resource, which is easy to achieve.
- It's is stateless, which means you don't need to worry about the surrounded context before you make a request for resources.
- Using Rest, we should make the response that we return to remain the same. In most cases, the response data should contain two parts: meta and data.
- Meta means the info about the request, is the request from client a success or a failure? Is it successfully understood by the server but the request is not allowed?
- Data means the resource that request want to get.
- In this project, the response are defined as follow:
{
"data": data,
"error_code": error_code,
"error_msg": error_msg,
}- For example, when a request want to get a user, let's say User 1, it may get:
{
"data": {
"id": 1,
"name": "John",
"web_site": "https://github.com/account",
"email": "hrui801@gmail.com",
"create_time": "2020-05-22 13:41:49",
"update_time": "2020-05-22 13:41:49"
},
"error_code": 0,
"error_msg": "success"
}- Something you should be aware of:
- Basically, we don't directly return JSON data in our handler function, instead we return an object.
- So before return to frontend we should marshal our object to JSON format.
- In flask, you can use jsonify function to do this, to transfer customized data to JSON, you can rewrite json.JSONEncoder function.
- In this project, the json.JSONEncoder is rewrite as:
class JsonEncoder(json.JSONEncoder):
def default(self, value) -> Any:
if isinstance(value, (datetime.datetime, datetime.date)):
return value.strftime("%Y-%m-%d %H:%M:%S")
if isinstance(value, ApiResponse):
return value.get()
if isinstance(value, BaseModel):
return value.marshal()
if isinstance(value, types.GeneratorType):
return [self.default(v) for v in value]
return json.JSONEncoder.default(self, value)- And then register it to flask app:
app.json_encoder = JsonEncoder- Finlay, before you return data to frontend, call
jsonifyfunction:
def schema(query_model: BaseQueryModel, response_model: ApiDataType):
def decorator(func):
@wraps(func)
def wrapper(self, **kwargs) -> Callable:
"""Some logic """
# jsonify function is called here
return jsonify(func(self, **kwargs))
return wrapper
return decorator- Exception doesn't mean bad, on the other hand, it's is crucial to let user know that they are doing something that the server can not satisfied. And they need to know why.
- In order to give the user corresponding error info, and to raise exception properly, this project uses a file exception to clarify all exceptions.
class ServerException(Exception):
code = 500
class ArgumentInvalid(ServerException):
code = 400- You may add a message property to each class.
- We want to raise exception in our code, a unified exception handler function is needed:
def handle_exception(e) -> Tuple[Dict[str, Union[Union[int, str, list], Any]], Union[int, Any]]:
code = 500
if isinstance(e, (HTTPException, ServerException)):
code = e.code
logger.exception(e)
exc = [v for v in traceback.format_exc(limit=10).split("\n")]
if str(code) == "500":
send_dingding_alert(request.url, request.args, request.json, repr(e), exc)
return {'error_code': code, 'error_msg': str(e), 'traceback': exc}, code- Then register to app:
app.register_error_handler(Exception, handle_exception)- Note: If you want to raise exception directly in your code and don't want to write exception handler function, all exceptions must be a subclass of werkzeug.exceptions.HTTPException.
- In object-oriented programming, it's better to keep your arguments to be a single object rather than many separated args. It's so in Python and Flask.
- Let's say you want to query a user by its name and/or age and/or email, it's better to write:
def get_user(filter_obj):
# filter_obj will have property:name, age, email
pass- not:
def get_user(name, age, email):
passclass BaseQueryModel(BaseModel):
def __init__(self, **kwargs: dict):
super().__init__(drop_missing=False, **kwargs)
""" Some logic """- And Query Mode's args Validation Can be write in BaseModel.
class BaseResponseModel(BaseModel):
def __init__(self, **kwargs):
super().__init__(**kwargs)
""" Some logic """- Don't forget to jsonify your response model before return to the frontend.
- Object-relational mapping is used to communicate with the database, it free you from writing SQL statements, keeps you coding in objected way.
from flask_sqlalchemy import SQLAlchemy
class Base(db.Model, SurrogatePK):
"""DataBase Model that Contains CRUD Operations"""
__abstract__ = True
"""Some logic that all subclass should be inherited """from sqlalchemy.ext.declarative import declarative_base
Meta = declarative_base()
db = SQLAlchemy(model_class=Meta)
class Base(db.Model, SurrogatePK):
"""DataBase Model that Contains CRUD Operations"""- Alembic is kind of like git, it keeps every version of your database model. It will generate a folder
alembic,alembic/versions, all versions are stored inversions.
- Swagger provides an automatic way to generate documentation, so you don't need to update your docs yourself. Web frontend developers can check it out for developing.
- But in my experience，fronted developer and backend developer discuss Product Requirements before developing, automatically generated docs are not specified enough to declare requirements. So this project don't use it.
- But it does no harm to use it, if you want to use swagger, check it out here.
- MVC(Model, View, Controller) is a typical design pattern. This project is programmed in MVC pattern, but is not strictly stick to it.
- Specifically, they are:
- Database Model, ORM is in this folder, which transfers your python object to database rows.
- QueryModel, arguments organized together as one Model. So frontend send args to backend, backend put them together to create a new object to do argument validation work, because use Model, some default functions can be bound to it.
- ResponseModel, resources that are returned to the frontend.
- Views are handling routing work, when a URL is requested, view function know what logic function to call.
- Handlers are used to do logic work, such as CRUD.
- folder exemptions store customized exception file and a send error to DingDing file.
- folder blueprints are used to organize all APIs.
- folder tests store all unit test case, using a test case is a good way to keep the project running well when a new feature is developed.
- Query one table 10 times each time to fetch one row is slow than query one table one time to fetch 10 rows.
- ORM release programmer from writing original SQL statements, but it also introduces some new problems. The Database Query N+1 Problem is a common one.
- Let's say you have two tables, User and Post, one user can write many posts. Consider the scenario below:
- you have a page that needs to show ten random users' info, and also you are asked to show all posts for each user.
- what will ORM do in this scenario? First, query User Table once to get ten users; Second, for loop users to query Post Table for each user at one time, ORM query Post Table ten times.
- code would be something like this:
users = User.query.limit(10) # query once
for user in users:
posts = Post.query.filter_by(user_id==user.id)
# Query ten times
# All times Query Table is 1 + 10- If you query User Table 1 time and get N users, then you need Query Post Table N times to get posts, all times query tables are 1+N, this is Called N+1 Problem(maybe called 1+N seems more reasonable).
- In fact, if you are familiar with join, there is no need to query N+1 times.
- So in your project, Carefully deal with this scenario.
- Facebook provides a solution called DataLoader to solve this.
- Although REST has many advantages, it does have some disadvantages.
- The major downside is a waste of resources, such as NetWork IO.
- In REST, you query a resource you get all fields of it. In many cases, you just want a part of it.
- Another problem is URL mixed, in REST, each resource need a URL, when there are many resources, managing these URLs could be a difficult problem.
- Because of all these, another framework Graphql is invented by Facebook.
- In Graphql, there is one URL, and you write queries to request the fields that you want to get. It fixes REST's problem.
- If you want to know more about graphql, check it out here.
- In this project, some code can be abstracted as a base class. In Handlers, basically they all did CRUD work, with only different args and slightly different logic.
- If CURD work has been abstracted as a base class, the code could be cleaner and simpler.
- Docker, using docker to run projects on different platforms.
- Docker-compose, a tool that provides you an easy way to manage docker.
- Makefile, auto compile tools that can save you a lot of time.
- pre-commit, auto-format you code before you commit, it keeps your project well-formatted, especially useful when working in teams.
- travis-ci.com, auto integration platform, which can run your tests automatically after you push.
- coveralls.io, get test coverage report after tests have been run, for example, you can use Travis-ci for CI and let it send report to coveralls.
- codebeat.co, calculate the complexity of your code.
- shields.io, provide beautiful metadata badge for your project. You can just simply put your GitHub repository URL in blank, it will automatically generate suggested badges for you.
- This project is coded in Python3.11 using flask-restful framework, you can treat it as a template to learn web-framework or just simply a start to use python writing web project.
- I try to explain the framework clearly above, but if you find any mistake or want to improve the code, you are welcomed to contact me at hrui835@gmail.com.
- If this project is helpful, please click a Star.
PRs are accepted.
Small note: If editing the README, please conform to the standard-readme specification.
MIT © 2020 ruicore