Recognize user-defined types in 'type:' attribute definition

Fixes the original complaint from #3431

Also:
* Update documentation
* Explain how to create new boxes in AGENTS.md
* Add a code coverage test

Author: ipspace

AGENTS.md

@@ -92,6 +92,11 @@ With the Box settings used in the topology data structure:
 * The interim dictionaries are created automatically: `a.b.c=1` works even when `a.b` does not exist yet.
 * The dotted paths can be used in indices (for example, `a['b.c'] = 1`) and in **in** tests (for example `'b.c' in a`)
 
+IMPORTANT -- the Box objects must have the default_box and box_dots flags set to work as expected:
+
+* When creating a new Box object from a dictionary, use data.get_box function
+* When creating an empty Box object, use data.get_empty_box function
+
 ### Using Jinja2 templates
 
 * The Jinja2 environment uses a custom `undefined` method that can handle dictionary hierarchy. For example, `a.b.c is defined` returns False instead of crashing even when `a.b` does not exist. There is no need for an extra `a.b is defined` guard.

docs/dev/validation.md

@@ -83,7 +83,7 @@ The dictionary-based approach allows in-depth validation of nested attributes, w
 (dev-valid-data-types)=
 ## Valid Data Types
 
-Validator recognizes standard Python data types (**str**, **int**, **float**, **bool**, **list** or **dict**) and the following networking-specific data types:
+Validator recognizes standard Python data types (**str**, **int**, **float**, **bool**, **list** or **dict**), the following networking-specific data types, and [user-defined data types](validate-user-types).
 
 | Data type.     | Meaning |
 |----------------|---------|
@@ -104,7 +104,7 @@ Validator recognizes standard Python data types (**str**, **int**, **float**, **
 | **r_proto**    | Routing protocol identifier |
 | **time**       | Time duration specified in seconds (`s`) or milliseconds (`ms`) |
 
-The data type can be specified as a string (without additional parameters) or a dictionary with a **type** attribute (data type as a string) and other type-specific validation parameters.
+The data type can be specified as a string (without additional parameters) or a dictionary with a **type** attribute (data type as a string) and other type-specific validation parameters. When specifying a user-defined data type with the **type** attribute, the definition of the user-defined data type is *merged* with the type definition.
 
 For example, VLAN link parameters include **access** and **native**, which can be any string value, as well as **mode** that must have one of the predefined values:
 
@@ -412,16 +412,6 @@ bgp:
       timers: exbs_timers
 ```
 
-Please note that the following definition (using `exbs_timers` as the value of `timers` **type** attribute) would not work:
-
-```
-bgp:
-  attributes:
-    global:
-      timers:
-        type: exbs_timers ### INVALID, WON'T WORK
-```
-
 You can use the `_namespace` attribute within the user-defined data types to add attributes from other objects. For example, as you can use link attributes in VLAN definitions, the **vlan** definition (see `modules/vlan.yml`) includes the `_namespace` attribute:
 
 ```

netsim/data/validate.py

@@ -7,7 +7,7 @@
 from box import Box
 
 from ..utils import log
-from . import get_a_list, get_empty_box
+from . import get_a_list, get_box, get_empty_box
 
 # We also need to import the whole data.types module to be able to do validation function lookup
 from . import types as _tv
@@ -395,6 +395,11 @@ def transform_validation_shortcuts(data_type: typing.Any) -> typing.Union[Box,di
 
   # Validating a dictionary against a dictionary of elements without a specified type
   if isinstance(data_type,Box):
+    if 'type' in data_type and isinstance(data_type.type,str):
+      if topo_attributes and data_type.type in topo_attributes:
+        base_type = transform_validation_shortcuts(topo_attributes[data_type.type])
+        data_type = get_box(base_type) + get_box({ k:v for k,v in data_type.items() if k != 'type' })
+
     if not 'type' in data_type:
       data_keys = { k:v for k,v in data_type.items() if not k.startswith('_') }
       data_type = Box({ k:v for k,v in data_type.items() if k.startswith('_') })

tests/coverage/errors/user-data-types.log

@@ -0,0 +1,8 @@
+IncorrectType in nodes: attribute 'nodes.b.bri' must be an integer or bgp_role_name, found str
+... use 'netlab show attributes node' to display valid attributes
+IncorrectValue in nodes: attribute nodes.f.brn has invalid value(s): cx
+... valid values are: provider, customer, peer, rs-server, rs-client
+IncorrectValue in nodes: attribute nodes.f.brx has invalid value(s): x
+... valid values are: a, b
+IncorrectType in nodes: attribute 'nodes.f.bri' must be an integer or bgp_role_name, found str
+Fatal error in netlab: Cannot proceed beyond this point due to errors, exiting

tests/coverage/errors/user-data-types.yml

@@ -0,0 +1,33 @@
+#
+# Code coverage test for user-defined data types used in "unusual" places
+#
+# Using user-defined data type in type: definition already works.
+# We still cannot use a user-defined data type as _alt_type
+#
+defaults.device: none
+
+defaults.attributes:
+  bgp_role_name:
+    type: str
+    valid_values: [ provider, customer, peer, rs-server, rs-client ]
+  node:
+    brn:
+      type: bgp_role_name
+    brx:
+      type: bgp_role_name
+      valid_values: [ a, b ]
+    bri:
+      type: int
+      _alt_types: [ bgp_role_name ]
+
+nodes:
+  a:
+    brn: provider               # Should succeed
+    brx: a                      # Should succeed
+    bri: 1                      # Should succeed
+  b:
+    bri: provider               # Should succeed
+  f:
+    brn: cx                     # Should fail (invalid value)
+    brx: x                      # Should fail (invalid value)
+    bri: x                      # Should fail (invalid value)