VTL 2.1 (Validation and Transformation Language, July 2024)

The Task Force for the Validation and Transformation Language (VTL), created in 2013 under the initiative of the SDMX Secretariat, has released the VTL version 2.1.

VTL 2.1 stems from the VTL 2.0 released in July 2018 and adds additional standard time operators, fixes bugs in the language specifications and takes account of feedback from users and execution engine implementers received since the 2.0.

A big change is going to be implemented in the documentation; to improve readability and maintainability the two manuals will be available also in a GitHub repository in markdown format, more examples will be added for each operator and the input/output structure of the operands and results will be added in a json representation; furthermore the data used in the examples will be available in csv format. By doing this, the overall quality of the documentation will be enhanced, since also the grammar will be derived as much as possible by the examples in the manuals. Finally a Technical Notes document will be available to facilitate developers of VTL tools (engines, interpreters, translators…) in the implementation.

The VTL 2.1 package includes:

  1. Part 1: the user manual, highlighting the main characteristics of VTL, its key assumptions and the information model the language is based on.
  2. Part 2: the reference manual (version July 2024), describing the full library of operators ordered by category, with examples. This version contains minor corrections (mainly typos) with respect to the version published in April 2018.

The grammar expressed in EBNF notation (Extended Backus-Naur Form), i.e. the technical notation for use in the implementation, will be added at the end of the revision process.

VTL 2.0 (Validation and Transformation Language, July 2018)

The Task Force for the Validation and Transformation Language (VTL), created in 2013 under the initiative of the SDMX Secretariat, has released the VTL version 2.0.

VTL 2.0 stems from the VTL 1.0 released in March 2015, from the draft 1.1 version released in October 2016 in public review and from the suggestions collected following such review. Due to the great improvements achieved, the new version has been named VTL 2.0 as a major release of the standard.

In designing VTL 2.0, a strong effort has been devoted to strengthen the fundamentals of the language, with the core objective of ensuring its stability in the future. A main demand originating from VTL implementers, in fact, is to protect the investments in developing VTL systems and construct a robust foundation, in order to ensure that future releases can leverage it and remain backward compatible. Great value is bestowed to the fact that current VTL 2.0 developments will work also with future releases.

VTL 2.0 provides important additional features compared to VTL 1.0. The capability of sharing and reusing definitions has been increased by means of the so-called “reusable rules” (for example validation, aggregation and transcoding rules). The possibility of defining and using custom operators (named user-defined operators) has been introduced: this is a main mechanism to achieve an extensible and customizable language.  A dedicated definition language supports the definition of reusable-rules and user-defined operators.  The manipulation language has been greatly improved and a number of new features have been added to the standard library, such as, for example, the aggregate and analytical operators.

The VTL 2.0 package includes:

  1. Part 1: the user manual, highlighting the main characteristics of VTL, its key assumptions and the information model the language is based on.
  2. Part 2: the reference manual (version July 2018), describing the full library of operators ordered by category, with examples. This version contains minor corrections (mainly typos) with respect to the version published in April 2018.
  3. The grammar expressed in EBNF notation (Extended Backus-Naur Form) which is the technical notation to be used in IT implementations (updated July 2020).

Download the VTL 2.0 package (user manual, reference manual, EBNF).

Further info: please contact the SDMX Technical Working Group (SDMX-TWG) at twg@sdmx.org.

Published on: 31 July 2020

VTL 1.1 (Validation and Transformation Language) in progress after public review

The Task force for the Validation and Transformation Language (VTL), created in 2013 under the initiative of the SDMX Secretariat, is working on version 1.1 of VTL. The public review version was released in October 2016 and was commented until January 2017. The consolidation of its final version is ongoing.

VTL is a standard language for defining validation and transformation rules (set of operators, their syntax and semantics) for any kind of statistical data. VTL 1.1 takes advantage of all the comments and proposals received after the publication of version 1.0, in March 2015. The new version contains several new features, together with an extended library of operators covering more use cases.

The initiative builds on the generic framework for defining mathematical expressions existing in the SDMX information model but the intention is to provide a language that is usable with other standards as well as for expressing logical validation rules and transformations on data.

The draft VTL 1.1 package includes:

  1. Part 1: the user manual, highlighting the main characteristics of VTL, its key assumptions and the information model the language is based on. Part 1 also includes a formal description of data validation objects against which information models can be mapped.
  2. Part 2: the reference manual, containing the full library of operators ordered by category, including examples; this version can support validation and basic compilation needs.

The eBNF (extended Backus-Naur Form), i.e. the technical notation for use in the implementation, will be added at the end of the revision process.

Download the draft VTL 1.1 package

For more info, please send a mail to the SDMX Technical Working Group (SDMX-TWG) at twg@sdmx.org.

Published on: 8 February 2017

VTL 1.0 (March 2015)

VTL is a standard language for defining validation and transformation rules (set of operators, their syntax and semantics) for any kind of statistical data. VTL builds on the Transformation section of the SDMX information model, by taking the common parts of GSIM, SDMX and DDI as for the representation of concepts and data. The assumption is that such a logical formalization of validation and transformation rules will provide a “technology-neutral” expression at business level of the processing taking place, against which various implementations and specific programming languages can be mapped. The specifications for exchanging VTL validation rules in SDMX messages, for storing rules and for requesting validation rules from web services will be provided in a specific update to the SDMX Technical Standards on which the SDMX Technical Working Group is working on. The VTL 1.0 package includes:

  1. Part 1, highlighting the main characteristics of VTL, its core assumptions and the information model the language is based on;
  2. Part 2, containing the full library of operators ordered by category, including examples; this first version can support validation and basic compilation needs. Future versions, whose design is already in progress, will include more features related to transformation of data;
  3. EBNF notation (Extended Backus-Naur Form) which is the technical notation to be used as a test bed for all the examples throughout the document.

Download the complete VTL 1.0 package (part 1, part 2, EBNF). Please send your comments to the SDMX Technical Working Group (TWG) at twg@sdmx.org.