Merge branch 'master' into openwrt-firewall

Author: jonathanunderwood

CHANGES.rst

@@ -1,7 +1,7 @@
 Change log
 ==========
 
-Version 0.9.0 [unreleased]
+Version 0.9.0 [2020-11-18]
 --------------------------
 
 - [change] **Potentially backward incompatible**:
@@ -10,6 +10,7 @@ Version 0.9.0 [unreleased]
   This change is backward incompatible if the same type of configuration
   was achieved using a workaround, in these cases the configuration
   will have to be upgraded to use the new format.
+- [feature] Added support for modem manager interfaces
 
 Version 0.8.2 [2020-08-17]
 --------------------------

docs/source/backends/openwrt.rst

@@ -294,6 +294,7 @@ There are 4 main types of interfaces:
 * **wireless interfaces**: must be of type ``wireless``
 * **bridge interfaces**: must be of type ``bridge``
 * **dialup interfaces**: must be of type ``dialup``
+* **modem manager interfaces**: must be of type ``modem-manager``
 
 Interface object extensions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1252,12 +1253,89 @@ Will be rendered as follows::
 
     package network
 
-    config interface 'xdsl'               
+    config interface 'xdsl'
             option ifname 'dsl0'
-            option proto 'pppoe'     
-            option username 'dsluser' 
+            option proto 'pppoe'
+            option username 'dsluser'
             option password 'jf93nf82o023$'
-            option mtu '1448'          
+            option mtu '1448'
+
+Modem Manager settings
+----------------------
+
+Interfaces of type ``modem-manager`` contain a few options
+that are specific to modem-manager interfaces (2G, 3G, 4G, LTE, etc).
+
+These are the ``OpenWrt`` backend NetJSON extensions for Modem Manager interfaces:
+
++--------------+---------+-----------------+--------------------------------------------+
+| key name     | type    | default         | allowed values                             |
++==============+=========+=================+============================================+
+| ``proto``    | string  | ``modemanager`` | ``modemanager``                            |
++--------------+---------+-----------------+--------------------------------------------+
+| ``apn``      | string  | empty           | APN, can be left blank                     |
++--------------+---------+-----------------+--------------------------------------------+
+| ``pin``      | string  | empty           | PIN code, can be left blank                |
++--------------+---------+-----------------+--------------------------------------------+
+| ``device``   | string  | empty           | path to device (see note below)            |
++--------------+---------+-----------------+--------------------------------------------+
+| ``password`` | string  | empty           | password, can be left blank                |
++--------------+---------+-----------------+--------------------------------------------+
+| ``username`` | string  | empty           | username, can be left blank                |
++--------------+---------+-----------------+--------------------------------------------+
+| ``metric``   | integer | ``50``          | metric, can be left blank                  |
++--------------+---------+-----------------+--------------------------------------------+
+| ``iptype``   | string  | ``ipv4``        | One of ``ipv4``, ``ipv6``, ``ipv4v6``      |
++--------------+---------+-----------------+--------------------------------------------+
+| ``lowpower`` | boolean | ``False``       | low power mode                             |
++--------------+---------+-----------------+--------------------------------------------+
+
+.. note::
+    ``device`` is a required property but can be left empty so that
+    the default value supplied by the hardware itself and already
+    present on the device can be left untouched by merging the
+    configuration generated with netjsonconfig
+    (instead of fully overwriting it).
+
+Modem Manager interface example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following *configuration dictionary*:
+
+.. code-block:: python
+
+    {
+        "interfaces": [
+            {
+                "type": "modem-manager",
+                "apn": "apn.operator.com",
+                "pin": "1234",
+                "device": "/sys/devices/platform/ahb/1b000000.usb/usb1/1-1",
+                "username": "user123",
+                "password": "pwd123456",
+                "metric": 50,
+                "lowpower": False,
+                "name": "modem0",
+                "mtu": 1500
+            }
+        ]
+    }
+
+Will be rendered as follows::
+
+    package network
+
+    config interface 'modem0'
+            option apn 'apn.operator.com'
+            option device '/sys/devices/platform/ahb/1b000000.usb/usb1/1-1'
+            option ifname 'wwan0'
+            option lowpower '0'
+            option metric '50'
+            option mtu '1500'
+            option password 'pwd123456'
+            option pincode '1234'
+            option proto 'modemmanager'
+            option username 'user123'
 
 Radio settings
 --------------

docs/source/general/contributing.rst

@@ -12,17 +12,13 @@ Follow these guidelines to speed up the process.
    :backlinks: none
    :depth: 3
 
-Reach out before you start
---------------------------
+General contributing guidelines of OpenWISP
+-------------------------------------------
 
-Before opening a new issue, try the following steps:
-
-- look if somebody else has already started working on the same issue
-  by looking in the `github issues <https://github.com/openwisp/netjsonconfig/issues>`_
-  and `pull requests <https://github.com/openwisp/netjsonconfig/pulls>`_
-- look also in the `OpenWISP mailing list <https://groups.google.com/d/forum/openwisp/join>`_
-- announce your intentions by opening a new issue
-- present yourself on the mailing list
+We highly recommend reading the general
+`OpenWISP Contributing Guidelines <https://openwisp.io/docs/developer/contributing.html>`_
+to find out the conventions we use to maintain consistency and quality standards
+across the different OpenWISP modules.
 
 Create a virtual environment
 ----------------------------
@@ -69,8 +65,8 @@ When you introduce changes, ensure test coverage is not decreased with:
 
     nose2 --with-coverage --coverage netjsonconfig
 
-Follow style conventions (PEP8, isort)
---------------------------------------
+Follow the coding style conventions and run the QA checks
+---------------------------------------------------------
 
 First of all, install the test requirements:
 
@@ -79,17 +75,18 @@ First of all, install the test requirements:
     workon netjsonconfig  # activate virtualenv
     pip install -r requirements-test.txt
 
-Before committing your work check that your changes are not breaking the style conventions with:
+Before committing your work, run ``openwisp-qa-format`` to format the code according
+to our `python code conventions <https://openwisp.io/docs/developer/contributing.html#python-code-conventions>`_:
 
 .. code-block:: shell
 
-    flake8
-    isort --check-only --recursive .
+    openwisp-qa-format
 
-For more information, please see:
+Then, you can run the QA checks with:
+
+.. code-block:: shell
 
-- `PEP8: Style Guide for Python Code <https://www.python.org/dev/peps/pep-0008/>`_
-- `isort: a python utility / library to sort imports <https://github.com/timothycrosley/isort>`_
+    ./run-qa-checks
 
 Update the documentation
 ------------------------

netjsonconfig/backends/openvpn/openvpn.py

@@ -110,7 +110,6 @@ def auto_client(
             'cert',
             'key',
             'pkcs12',
-            'mtu_disc',
             'mtu_test',
             'fragment',
             'mssfix',

netjsonconfig/backends/openwrt/converters/interfaces.py

@@ -2,6 +2,7 @@
 from copy import deepcopy
 from ipaddress import ip_address, ip_interface
 
+from ..schema import schema
 from .base import OpenWrtConverter
 
 
@@ -109,6 +110,16 @@ def __intermediate_interface(self, interface, uci_name):
             del interface['wireless']
         if 'addresses' in interface:
             del interface['addresses']
+        # specific transformation
+        type_ = self._get_uci_name(interface["type"])
+        method = getattr(self, f'_intermediate_{type_}', None)
+        if method:
+            interface = method(interface)
+        return interface
+
+    def _intermediate_modem_manager(self, interface):
+        interface['proto'] = 'modemmanager'
+        interface['pincode'] = interface.pop('pin', None)
         return interface
 
     _address_keys = ['address', 'mask', 'family', 'gateway']
@@ -225,6 +236,10 @@ def __netjson_interface(self, interface):
             interface['mac'] = interface.pop('macaddr')
         if interface['network'] == self._get_uci_name(interface['name']):
             del interface['network']
+        # specific transformation
+        method = getattr(self, f'_netjson_{interface.get("proto")}', None)
+        if method:
+            interface = method(interface)
         return interface
 
     def __netjson_type(self, interface):
@@ -243,13 +258,17 @@ def __netjson_type(self, interface):
         return 'ethernet'
 
     def __netjson_addresses(self, interface):
-        proto = interface.pop('proto', 'none')
+        proto = interface.get('proto', 'none')
+        address_protos = ['static', 'dhcp', 'dhcpv6', 'none']
+        if 'proto' in interface and proto in address_protos:
+            del interface['proto']
         if 'ipaddr' not in interface and 'ip6addr' not in interface and proto == 'none':
             return interface
-        if proto not in ['static', 'dhcp', 'dhcpv6', 'none']:
-            interface['proto'] = proto
-            interface['type'] = self.__get_special_interface_type(interface)
+        if proto not in address_protos:
+            interface['type'] = 'other'
+        return self._add_netjson_addresses(interface, proto)
 
+    def _add_netjson_addresses(self, interface, proto):
         addresses = []
         ipv4 = interface.pop('ipaddr', [])
         ipv6 = interface.pop('ip6addr', [])
@@ -273,14 +292,19 @@ def __netjson_addresses(self, interface):
             interface['addresses'] = addresses
         return interface
 
-    def __get_special_interface_type(self, interface):
-        username = interface.get('username', False)
-        password = interface.get('password', False)
+    def _netjson_dialup(self, interface):
+        interface['type'] = 'dialup'
+        return interface
+
+    _modem_manager_schema = schema['definitions']['modemmanager_interface']
 
-        if username and password:
-            return 'dialup'
+    def _netjson_modem_manager(self, interface):
+        del interface['proto']
+        interface['type'] = 'modem-manager'
+        interface['pin'] = interface.pop('pincode', None)
+        return self.type_cast(interface, schema=self._modem_manager_schema)
 
-        return 'other'
+    _netjson_modemmanager = _netjson_modem_manager
 
     def __netjson_address(self, address, interface):
         ip = ip_interface(address)
@@ -308,7 +332,7 @@ def __netjson_parse_ip(self, ip, netmask=32):
         if ip and netmask:
             return '{0}/{1}'.format(ip, netmask)
         else:
-            None
+            return None
 
     def __netjson_dns(self, interface, result):
         key_mapping = {'dns': 'dns_servers', 'dns_search': 'dns_search'}
@@ -320,3 +344,19 @@ def __netjson_dns(self, interface, result):
                 items = items.split()
             result.setdefault(netjson_key, [])
             result[netjson_key] += items
+
+
+for proto in [
+    '3g',
+    '6in4',
+    'aiccu',
+    'l2tp',
+    'ncm',
+    'ppp',
+    'pppoa',
+    'pppoe',
+    'pptp',
+    'qmi',
+    'wwan',
+]:
+    setattr(Interfaces, f'_netjson_{proto}', Interfaces._netjson_dialup)

netjsonconfig/backends/openwrt/schema.py

@@ -7,6 +7,9 @@
 from .timezones import timezones
 
 default_radio_driver = "mac80211"
+_interface_properties = default_schema["definitions"]["interface_settings"][
+    "properties"
+]
 
 # The following regex will match against a single valid port, or a port range e.g. 1234-5000
 port_range_regex = "^([1-9][0-9]{0,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])(-([1-9][0-9]{0,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5]))?$"  # noqa
@@ -151,6 +154,7 @@
                             },
                             "proto": {
                                 "type": "string",
+                                "title": "protocol",
                                 "enum": [
                                     "3g",
                                     "6in4",
@@ -165,7 +169,7 @@
                                     "wwan",
                                 ],
                                 "default": "pppoe",
-                                "propertyOrder": 8,
+                                "propertyOrder": 1.1,
                             },
                             "username": {
                                 "type": "string",
@@ -182,7 +186,53 @@
                     {"$ref": "#/definitions/interface_settings"},
                 ],
             },
-            "base_radio_settings": {  # Overrides default schema
+            "modemmanager_interface": {
+                "type": "object",
+                "title": "Modem manager interface",
+                "required": ["name", "device"],
+                "properties": {
+                    "name": _interface_properties["name"],
+                    "mtu": _interface_properties["mtu"],
+                    "autostart": _interface_properties["autostart"],
+                    "disabled": _interface_properties["disabled"],
+                    "type": {
+                        "type": "string",
+                        "enum": ["modem-manager"],
+                        "default": "dialup",
+                        "propertyOrder": 1,
+                    },
+                    "apn": {"type": "string", "title": "APN", "propertyOrder": 1.1},
+                    "pin": {
+                        "type": "string",
+                        "title": "PIN code",
+                        "propertyOrder": 1.2,
+                    },
+                    "device": {
+                        "type": "string",
+                        "description": "Leave blank to use the hardware default",
+                        "propertyOrder": 1.3,
+                    },
+                    "username": {"type": "string", "propertyOrder": 1.4},
+                    "password": {"type": "string", "propertyOrder": 1.5},
+                    "metric": {"type": "integer", "default": 50, "propertyOrder": 1.6},
+                    "iptype": {
+                        "type": "string",
+                        "title": "IP type",
+                        "default": "ipv4",
+                        "enum": ["ipv4", "ipv6", "ipv4v6"],
+                        "options": {"enum_titles": ["IPv4", "IPv6", "IPv4 and IPv6"]},
+                        "propertyOrder": 1.7,
+                    },
+                    "lowpower": {
+                        "type": "boolean",
+                        "title": "Low power mode",
+                        "format": "checkbox",
+                        "default": False,
+                        "propertyOrder": 1.8,
+                    },
+                },
+            },
+            "base_radio_settings": {
                 "properties": {
                     "driver": {
                         "type": "string",
@@ -236,7 +286,12 @@
                 }
             },
             "interfaces": {
-                "items": {"oneOf": [{"$ref": "#/definitions/dialup_interface"}]}
+                "items": {
+                    "oneOf": [
+                        {"$ref": "#/definitions/dialup_interface"},
+                        {"$ref": "#/definitions/modemmanager_interface"},
+                    ]
+                }
             },
             "routes": {
                 "items": {