models_serializers_viewsets.md 12.4 KB
Newer Older
1 2 3 4 5 6
Models, Serializers and ViewSets
=============

## Basics

Our backend app is based on Django. If you are familiar with Django then you will understand what is a Model.
7
For this project, the general idea is that Django is used to handle the data and nothing else. Lets say that we are only using the "M" of "MVC" of a standard Django architecture.
8 9 10 11 12 13 14 15

To be able to transfer data to the frontend, we use the famous Django extension [django-rest-framework](https://www.django-rest-framework.org/).

?> Make sure to look at its documentation and code if you want to now more about it.


To put it into a nutshell, we have:

16 17 18 19 20
- **Models** that handle all data and the exchanges with the database,
- **Serializers** that handle the conversion from models' data to JSON and from JSON to models' data,
- **ViewSets** that handle communication with the frontend.


21 22
## In details

23
Since the process of creating the API was a bit repetitive, we have used a lot of [inheritance](Application/Backend/architecture.md).
24 25 26 27 28

Below are the things to know about what has been implemented.

### Models

29
There exist different *abstract* models from which the *concrete* models can inherit from. Those *abstract* models have different behaviors and properties built in directly.
30

31 32 33 34 35 36 37 38 39
- `BaseModel`: All models from the app inherit from this simple one. It should be directly used only for data that will be automatically stored and not editable.
- `EssentialModule`: contains attribute that should be inherited from by all models that corresponds to frontend editable modules. It manages the moderation aspect.
- `VersionedEssentialModule`: inherits from `EssentialModule` and make it possible to version the data contained in the models that inherit from it.
- `Module`: inherits from `VersionedEssentialModule` and provides a set of attributes such a title, a comment, a custom useful links field and an importance level.

Here is the UML of their inheritance:

![abstract](../../generated/abstract.svg)

40

41 42 43
?> :information_desk_person: To register your model in the app, you need to import it in `backend/backend_app/admin.py` and add it to the `ALL_MODELS` list.

------
44

45
To define a model for the app, you should (in most cases) use one of the three base classes defined above. What remains is to define a model just like you would do for any Django Model. Here is an example (you have tons of them in app already):
46 47 48

```python
from django.db import models
49
from backend_app.models.abstract.base import BaseModel
50

51
class Country(BaseModel):
52 53 54 55 56 57 58 59 60 61 62 63 64
    name = models.CharField(max_length=200)
    iso_alpha2_code = models.CharField(primary_key=True, max_length=2)
    iso_alpha3_code = models.CharField(
        unique=True, max_length=3, default="", blank=True
    )
    region_name = models.CharField(max_length=200)
    region_un_code = models.CharField(max_length=3)
    sub_region_name = models.CharField(max_length=200, default="", blank=True)
    sub_region_un_code = models.CharField(max_length=3, default="", blank=True)
    intermediate_region_name = models.CharField(max_length=200, default="", blank=True)
    intermediate_region_un_code = models.CharField(max_length=3, default="", blank=True)
```

65 66 67
:warning: Also, our use of models in the app differs from Django on one aspect. To handle different moderation settings, there is one extra **optional** attribute you can use in your model which is `moderation_level`. It can either be `0`, `1` or `2` (default value). You can find more about this parameter [here](Application/Backend/moderation_and_versioning?id=model-level).


68 69 70
### Serializers


71
To go along the previously mentioned *abstract* models, we have matching serializers that handle all the logic behind rendering the models' field to JSON, validating the data before creating/updating a model instance and handling the specific behaviors (such as moderation — that comes with all models that inherit from `EssentialModule` — and versioning — that comes with all models that inherit from `VersionedEssentialModule`).
72 73 74 75 76 77 78

:warning: **95% of the *magic* behind those not that simple behaviors are handled in a subtle, sometimes *hacky*, manner.**

:information_desk_person: All those specific behaviors should have been correctly unit-tested at this point.

?> You can find more information about moderation and versioning [here](Application/Backend/moderation_and_versioning.md).

79
----
80

81
To create a serializer for your model, you need to create a class that inherits from the matching parent class of your model. For instance the `Country` model inherits from the `BaseModel` *abstract* class, so the `CountrySerializer` inherits from `BaseModelSerializer`.
82

83
Then you must define a `Meta` class inside the serializer that contains information about what model is the serializer for and what fields (attribute) should be taken into account. For instance for the country model we have the following serializer:
84 85

```python
86
class CountrySerializer(BaseModelSerializer):
87 88 89 90 91 92 93 94 95 96
    class Meta:
        model = Country
        fields = "__all__"
```

So the serialization of all fields will be handled automatically by `django-rest-framework`.

You may also decide to exclude some fields:

```python
97
class UniversitySerializer(BaseModelSerializer):
98 99 100 101 102 103 104 105
    class Meta:
        model = University
        exclude = ("utc_id",)
```

And even include custom serializer to make some fields read only, or add data to send to the client:

```python
106 107 108 109 110
class BaseModelSerializer(MySerializerWithJSON):
    """
    Serializer to go along the BaseModel model. This serializer make sure some
    relevant data is always returned.
    """
111 112 113 114 115

    # For easier handling on the client side, we force an id field
    # this is useful when a model has a dedicated primary key
    id = serializers.SerializerMethodField()

116
    def get_id(self, obj: BaseModel):
117 118 119 120 121 122
        """
        Serializer for the id field.
        """
        return obj.pk

    class Meta:
123
        model = BaseModel
124 125 126 127 128 129
    # ...
```

As shown above, when using a `custom_attr = serializers.SerializerMethodField()`, you need to define a method in the class that is named `get_custom_attr` and returns whatever you want (as soon as it can be converted to JSON).


130
?> Django-rest-framework is full a functionalities, and only a subset of them have been presented here. So feel free to have a look at [the package documentation](https://www.django-rest-framework.org/).
131 132 133 134


### Viewsets

135
The custom work performed on the generic viewsets is fairly straightforward, you can have a look at it. Anyway, just like with the serializer, you should use the corresponding viewset when inheriting: `BaseModelViewSet` or `EssentialModuleViewSet` or `VersionedEssentialModuleViewSet` or `ModuleViewSet`.
136

137
All the parametrization of the ViewSets happens within them.
138

139
Just like in the standard use of the Django Rest Framework, you should set the following attributes:
140

141 142 143
* `queryset` (or `get_queryset`): the queryset associated with the viewset,
* `serializer_class`: what is the serializer class to use to serialize the queryset.
* `viewset_permission` (defaults to `(IsAuthenticated & (IsStaff | NoDelete))`): what are the permissions associated with each viewset. If something is specified then it will be composed (`&`) with the default one. You can compose permissions with `&` (logical *and*) or `|` (logical *or*). Most permissions are defined in `backend/backend_app/permissions/app_permissions.py`, you can add yours too.
144
* `filterset_fields` (optional): more information [here](Application/Backend/models_serializers_viewsets?id=filtering)
145

146 147 148 149 150 151 152 153 154 155 156 157 158
?> :information_desk_person: If you set `viewset_permission`, it must be a tuple; (in our case it will often have only one attribute).

!> Unlike with a standard use of the Django Rest Framework, you must set the route of the endpoint for the viewset directly within the viewset (it is used for registering the endpoint in the router and in other locations).

To do so, you must add the `end_point_route` attribute to your ViewSet class (e.g.: "countries"; `api/` will be automatically prepended). On this attribute you should add any group you would like to capture (eg: `myEndPoint/(?P<content_type_id>[0-9]+)/`). More about this in few lines.

?> :information_desk_person: The `end_point_route` string will also be used for naming variables in JS, so keep it simple and consistent please. :smile:

!> As a result, **if you change an `api_end_pont` you will most likely need to update some JS files (see [here](Application/Frontend/redux.md)).**

-----

Most often a viewset class will look like this:
159 160

```python
161

162 163 164
class CountryViewSet(BaseModelViewSet):
    queryset = Country.objects.all()  # pylint: disable=E1101
    serializer_class = CountrySerializer
165 166
    permission_classes = (ReadOnly,)
    end_point_route = "countries"
167 168
```

169 170 171 172 173 174 175
#### Filtering

Sometimes you will also need to filter the data against some attribute found in the client's request.

You should favor the standard approach to do so. If the endpoint is `/api/endpoint` and you want to make sure the model attribute `attr` equals `x` for the object returned then the client should fetch the following route: `/api/endpoint?attr=x` (if multiple attributes need to be filtered then it would look like `/api/endpoint?attr1=x&attr2=y&attr3=z`).

To make it work like this, you must add `filterset_fields` to your viewsets. It should be a tuple of the names of the fields you allow filtering on.
176 177

```python
178
class CountryDriViewSet(ModuleViewSet):
179 180
    queryset = CountryDri.objects.all()  # pylint: disable=E1101
    serializer_class = CountryDriSerializer
181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200
    permission_classes = (IsStaff | IsDri | NoPost,)
    end_point_route = "countryDri"
    filterset_fields = ("countries",)
```

This filtering is achieved through [`django-filter`](https://django-filter.readthedocs.io/en/master/) package (and the [standard api provided by `django-rest-framework`](https://www.django-rest-framework.org/api-guide/filtering/)), you can further customize the filtering if you'd like to.

------

There exist an other way of filtering directly on the client request, **but you should use it only when there are absolutely no other alternatives or when parameters are absolutely required**. You can see this other way below:

* Endpoint's attributes are set to be captured in `end_point_route`,
* We make use of them in `get_queryset`.

```python
class VersionViewSet(BaseModelViewSet):
    # ...
    end_point_route = (
        r"versions/(?P<content_type_id>[0-9]+)/(?P<object_pk>[0-9A-Za-z]+)"
    )
201 202

    def get_queryset(self):
203 204 205 206 207 208
        content_type_id = self.kwargs["content_type_id"]
        object_pk = self.kwargs["object_pk"]
        ct = ContentType.objects.get_for_id(content_type_id)
        model = ct.model_class()
        obj = model.objects.get(pk=object_pk)
        return Version.objects.get_for_object(obj)
209 210
```

211 212 213 214 215 216

#### Performance

!> **Viewsets can be huge bottlenecks for performances.**

If the objects the viewset will be serializing have a lot of "foreign" data (from a ForeignKey, ManyToMany field, etc.), you **shall** extend the queryset to make sure to prefetch all the data needed for the serializer:
217 218 219

```python
# Extract of the class definition
220 221 222
class EssentialModuleViewSet(BaseModelViewSet):
    serializer_class = EssentialModuleSerializer

223 224 225 226 227 228 229 230
    def get_queryset(self):
        """
        Extended default rest framework behavior
        to prefetch some table and enhance performances
        """
        return self.queryset.prefetch_related("moderated_by", "updated_by")
```

231
?> :information_desk_person: This will basically generate an SQL `JOIN` once per queryset. If you don't do this, the serialization of each attribute will result to one `SQL` query, leading to hundreds of them on a simple client request.
232 233 234 235 236 237 238

## Keeping things consistent

!> :warning: To keep things consistent the general idea is that if a model is called `Blabla` than it's serializer class will be named `BlablaSerializer` and the associated viewset class will be named `BlablaViewSet`.

:information_desk_person: Sometimes it will be useful to have different viewsets for the same serializer, for instance if you want an api endpoint that returns the data filtered in a certain way.

239 240
!> You should avoid having multiple serializers for the same model, especially if that model is versioned.

241
In fact, to deserialize the versions, we try to infer the correct serializer to use in `backend/backend_app/models/version.py`. **In case multiple serializers point to same model, you must define the `get_serializer` class method in your model:**
242 243 244 245 246 247 248 249

```python
class MyAwesomeModel(VersionedEssentialModule):
    @classmethod
    def get_serializer(cls):
        return MyAwesomeModelSerializer
    ...
```