In this article I’ll explain how to design and develop, following a Design-First approach, a REST API to enable access to Environmental Monitoring data.
In my previous article on Medium, entitled “A Design-first approach to API implementation”, I showed, with a fair level of detail, how to design a REST API using Oracle Apiary and, then, how to use the interface definition, based on OpenAPI (Swagger 2.0), in order to generate, automatically, the initial “skeleton” for the implementation of the service, based on Python-Flask.
Apiary is an acquisition made by Oracle, integrated into the Oracle API Platform Cloud Service. An important element for designers and developers, in my opinion, is that Oracle has retained the ability to try and use Apiary for free. It is even possible to sign-up using your Github account.
In this article I start from the general description made in the previous article and I show a concrete realization: how to realize the first core of a REST API that allows access to the measurements made by a set of devices containing sensors, which measure temperature, humidity, but also data on air quality and pollution (pm2.5, pm10, etc). The devices could be placed, for example, in a large building (the headquarter of a company) or in a shopping center.
The concept from which I started, applied to Smart Cities, in summary is that to maximize the benefits that the adoption of technology can bring to Citizen, it is important to make available the data collected to a wide audience, in an open format, so that everyone can consult but also to allow the creation of an ecosystem of solutions, App, Chatbot and other tools that facilitate access to data.
Some details for the implementation..
The data collected by the sensors are stored in an Oracle MySQL database. One of the best choices, also taking into account the results of the last IoT Developer Survey held by the Eclipse Foundation.
The important portion of data, for the purpose of this article, is in a single MySQL table where:
- each device (box containing a group of sensors) is identified by a numeric "device_id" (id_list)
- Each record contains a "timestamp" (Unix timestamp, 10 digits)
- The record contains temporally related measures (theoretically same instant)
The functionalities we want to implement are, for now, only two, the initial core of a more complete realization:
- Read last measure (available, for device #N)
- Read last M measures (for device #N)
The serialization format chosen for the messages is JSON.
Apiary based approach on Design.
In Apiary, after creating a project, the first important operation to be performed is the definition of the API interface. The tool is the Editor.
As specification language you can use OpenAPI (Swagger) or Blueprints API. I have decided, for obvious reasons as you will see, to adopt the first one.
In the editor the screen is divided in two parts: on the left the definition of the interface in Swagger 2.0, on the right, automatically, is generated the documentation and a menu for the test.
The language for the specification is quite simple. In summary we must define:
- The name of the project and the version of Swagger (in our case 2.0)
- A description, useful for documentation
- The base path (basePath) for URLs
- The paths (path) for each function and the HTTP (GET ...) verbs allowed
For each operation:
- (possible) input parameters
- response format
An example can also be provided for the answer. This example is used by the Mock Server that Apiary makes available.
The data structures used in the answers are defined in the "definitions" section. In them we can define the type and format (ex: int64) of the data.
For data specification see: Data Types-Swagger
The interface specification is shown in the following Gist:
“Mock Server” and specification sharing.
In an agile, modern approach, it is advisable to make available the interface specification as soon as it is defined and approved, so that developers, for example for Apps or Chatbot, who need to use it have a certain and authoritative reference point.
Furthermore, it is very convenient for developers to have a "Mock Server", which simulates the service, so that they can test the interaction between APIs and clients being developed.
In this regard, Apiary offers two very important features:
- a Mock Server that can be invoked; It obviously returns the examples defined in the specification
- Automatic generation of client code, for various languages
Generation of Server-side implementation.
For the implementation of the REST API I will once again follow the approach adopted in the previously mentioned Medium article.
In summary, the starting point is the swagger.yaml file provided and validated by Apiary, which contains the interface specification (see GIST above).
The tool to be used for generating the code is swagger-codegen, with the command:
swagger-codegen generate -i ./swagger.yaml -l python-flask
To access the MySQL database I decided to adopt the "Connector/Python" module.
Additional Python modules must be added to the requirements.txt file
and installed with:
pip install -r requirements.txt
“Connector/Python” is very effective as it also allows to implement Connection Pooling.
Having chosen Python and Flask, the code generated is based on the framework Connexion.
If we explore the code, inside swagger_server directory, we find a subdirectory containing the Model and a subdirectory containing the Controller.
The file we must modify in order to complete our implementation, with the code handling access to MySQL DB, is default_controller_py.
Here the complete code I have developed:
At the end it is possible to start the server with the command (see also the instructions contained in the README generated by swagger-codegen):
python3 -m swagger_server
By default, the server will listen on port 8080.
Some best practices and recommendations.
The code shown is for a prototype that wants to "indicate" how to proceed. It is not entirely suitable for use in a production environment.
However, some "best practices" have been adopted and/or identified:
- Use a Connection Pooling: creating connections to the DB is always an "expensive" and "slow" operation, so avoid creating a connection for each request, but use instead a connection pool; The createConnection() function shows this approach
- Do not insert the credentials for opening the connections in the main Python code, but place them in a separate configuration file (in our case dbconfig.py)
- Use a MySQL account for data access that has only READ privilege on the tables
- Flask has its own HTTP server; It is not appropriate to use it in a production environment. You can use NGINX as a Reverse Proxy, or there are other alternatives (WSGI)
Advantages of a Design-First approach, based on Apiary.
In conclusion, we have shown how, starting from existing data produced by sensors and stored in a MySQL database, it is quite easy to create a REST API that exposes and facilitates access to such data.
The approach based on Apiary, Design-First, allows you to start focusing only on the interface, as seen by the clients and abstracting completely from the implementation.
But, the use of Apiary also makes available immediately, in Cloud, the definition of the interface, the documentation and a Mock Server to support the development of the clients. All synchronized.
Finally, combining Apiary with the swagger-codegen tool, we have seen how starting from the YAML file that defines, in OpenAPI 2 format, the interface, the implementation of the server skeleton is fast.
In this article we have addressed an important topic, security, only in a minimal form.
Obviously, an API exposed on the Internet must be adequately protected, even just to avoid “denial-of-service” attacks.
For example, to allow access only to registered applications, a mechanism based on "API KEYS" can be adopted. This does not mean restricting who can access, but more simply and reasonably to keep track of who requires access, and eventually be able to block those who make use of the API contrary to the rules defined.
Obviously, security must be implemented not in the code but in a structural way. All security and management features can be implemented using the other components of the Oracle API Platform Cloud.
But this is the subject, maybe, of another article. In the meantime I could recommend reading this one: https://redthunder.blog/2018/01/29/teaching-how-to-design-and-secure-an-api-with-oracle-api-platform/
Almost nothing is born in the desert. This article wouldn’t have been possible without support and suggestions from the manager and all the colleagues of the team I work in: Oracle Italy Innovation Solutions Team. Also, I must say a big thanks to the Partner and all the people from Harpa Italia, who have provided the devices (Aircare) and the core SW infrastructure, hosted for the joint solution in Oracle Cloud Infrastructure. Last but not the least, this work has been developed as part of the effort to build the ProximaCity Demo: a demo of a Smart City. Have a look.