Roslyn has been known as the code name for the next generation of C# compiler, at least since its first public preview was released in 2011. Infact the project started internally at Microsoft a couple of years earlier. Even before its first final release in Visual Studio 2015, it started to mean a lot more than just a new compiler. At that time, it also got a new official name: .NET Compiler Platform. Nevertheless, the word Roslyn is still a part of developer vocabularies and will probably remain in use for quite some time. Let us look at what one might be referring to today when mentioning Roslyn, what the current state of the project is, and how it can be beneficial to developers.
This article is published from the DNC Magazine for .NET Developers and Architects. Download this magazine from here [Zip PDF] or Subscribe to this magazine for FREE and download all previous and current editions
.NET Compiler Platform (a.k.a Roslyn)
Most developers treat compilers as black boxes: they receive source code as input, do some processing on it, and output executable binaries. Computer science graduates, who have taken a compiler course during their studies, might remember that internally compilers have a pipeline structure. Source code processing consists of three main phases: syntax analysis, semantic analysis, and code generation. Although each phase outputs its own intermediate results and passes them as input to the next phase, these are just internal data structures that are not accessible outside the compiler.
Image 1: Compiler as a black box
In the early days of software development, when source code at its very best was written in simple text editors, and compilers were invoked from the command line, this sufficed. However, throughout the years, our expectations have changed a lot. Modern integrated development environments, such as Visual Studio, offer so much more: syntax highlighting, code completion, debugging, refactoring, static analysis, etc. Even in simple programming text editors, such as Sublime Text, we expect at least syntax highlighting and code completion.
Most of these services require some level of knowledge about source code. Since compilers do not share the intermediate results of their processing, the editors need to repeat parts of the same work. For example, the editor in Visual Studio 2013 and earlier versions, did a lot of C# language processing independently of the C# compiler. Not only that; even their code bases were separate, potentially causing them to behave differently.
When Microsoft decided to rewrite the C# and Visual Basic compilers from scratch (because their code base became a challenge to maintain and adding new features became unfeasible), they did not want to improve just the code. They rather wanted to make it useful in other scenarios as well: diagnostics, static analysis, source code transformation, etc. In order to achieve that, Microsoft created a compiler that not only converts source code into binaries, but also acts as a service, providing a public API for understanding the code.
Roslyn Public API
Roslyn APIs reflect the pipeline architecture of a traditional compiler, giving access to each step of compiler’s source code analysis and processing:
- Syntax tree API exposes the lexical and syntactic structure of the code, including formatting and comments. The latter two are irrelevant for later phases of the compiler pipeline, but important for tools that want to manipulate the code and keep the formatting and code comments intact.
- Symbol API exposes the symbol table containing names declared in the source code, as well as those originating from referenced assemblies without corresponding source code.
- Binding and Flow Analysis APIs exposes the complete semantic model of the code that becomes available after the binding phase. These APIs contain all the information about the code that is required for generating the binaries, including any errors and warnings that were detected in the code.
- Emit API provides access to services for emitting the IL byte code.
Image 2: Compiler as a service
The scope of Roslyn is not limited to the compiler API. Most of its integration into Visual Studio 2015 is part of the public API as well, except for the thin layer interacting with its proprietary code:
- Diagnostic API enables development of custom third party code diagnostics that are pluggable into Visual Studio code editor and MSBuild build system, making them indistinguishable from compiler’s built-in diagnostics for detecting errors and warnings in code.
- Scripting API provides the interactive C# environment in Visual Studio 2015 Update 1 – the so-called REPL (read-eval-print loop).
Image 3: C# interactive in Visual Studio 2015 Update 1
- Workspaces API provides the model of a complete solution, consisting of multiple projects with documents and assembly references, making it possible to drill down all the way to the semantic and syntactic models of the code.
Roslyn code base is now being used in two scenarios with almost opposite requirements. For the compiler, high throughput and correctness are most important; whereas for the interactive editor, responsiveness and error tolerance are of higher priority. Roslyn managed to achieve comparable, if not better performance than Visual Studio 2013, in both fields. However, this required certain compromises that are evident from the API. All of the data structures (syntax trees, semantic model, and workspaces) are immutable; i.e. they cannot be modified.
Every time a developer changes a single character in any of the files, a new copy of all the data structures is created, leaving the previous version unchanged. This allows a high level of parallelism and concurrency in the Roslyn engine, as well as in its consumers, thereby preventing any race conditions to occur. Of course, in the interest of performance, these operations are highly optimized and reuse as much of the existing data structures as possible. Again, those being immutable makes this possible!
Immutability of data structures affects API consumers in a significant manner:
- None of the models exposed by the API can be modified. Instead, they provide methods for creating copies with a specific change applied to them.
- Since no data structure ever changes, events make no sense. Instead, user code can register to be called when a new version of the model is created, and the models can be compared to detect the differences.
Immutability does require some getting used to; but in the end, it does not have a negative effect on what can be achieved using the API. After all, C# and Visual Basic support in Visual Studio 2015 is completely based on the same public API.
Roslyn in Action
As mentioned above, Visual Studio 2015 is taking full advantage of .NET Compiler Platform, also known as Roslyn. Many of the code editor functionalities are implemented using Roslyn public API: from automatic code formatting and coloring; to IntelliSense, code navigation and refactoring.
Since the full public API is available as open source, other integrated development environments are already switching from their internal implementations, to using Roslyn as the engine powering the advanced code editor features. At the time of writing, preview versions of Xamarin Studio and Mono Develop are available with .NET compiler platform already integrated.
To bring the power of Roslyn to other text editors that do not run on .NET and cannot host Roslyn in process, OmniSharp project has been brought to life. It is a cross-platform wrapper around Roslyn, exposing its API across HTTP. This makes it accessible to any text editor, and there are already integrations available for many popular programming text editors of today: Atom, Brackets, Sublime Text, Visual Studio Code, and others.
With the help of Roslyn, code diagnostics and refactoring became accessible to individual developers. This resulted in several new specialized extensions for code analysis, the most popular of them being Refactoring Essentials, Code Cracker, and CSharp Essentials. They are free and do a lot of what was once only possible using large expensive commercial extensions like ReSharper, CodeRush, and JustCode. It will be interesting to see, how these extensions will adapt to the changes. DevExpress has already released a preview of CodeRush for Roslyn – a rewrite of the original CodeRush extension. JetBrains has announced that they do not plan to use Roslyn in future versions of ReSharper, while Telerik as of this writing, has not shared any news about their plans for Just Code.
Of course, Roslyn is not limited to being useful only in code editors. Standalone tools that need to process C# or Visual Basic code can benefit as well. The two most prominent open source projects in this field are scriptcs and Source Browser. The former makes C# a scripting language, allowing simple scripts written in C# to be executed directly from command line. The latter is powering Microsoft’s reference source web site – the place to browse .NET source code online with full syntax highlighting, navigation and search. Source Browser project can be used to host any other .NET project source code online in a similar manner.
Image 4: .NET Framework Reference Source web site
This is far from an exhaustive list of publicly available projects using Roslyn, let alone proprietary ones being developed for internal use only. Still, they clearly show that the possibilities opened up by public Roslyn APIs are often limited only by imagination.
Getting Engaged with Roslyn
The most obvious way to benefit from the Roslyn open source project is probably by using its APIs to develop custom code diagnostics, refactoring or standalone tools. Two of my articles that can help you get started have already been published in previous editions of the DotNetCurry (DNC) magazine: Diagnostic Analyzers in Visual Studio 2015 and Create Your First Diagnostic Analyzer in Visual Studio 2015. There is also a lot of documentation and samples on the official Roslyn open source site. In spite of that, development with Roslyn is not trivial, and it might not seem very applicable to your daily work.
However, there are other ways to get involved in Roslyn open source community. Having all the source code available allows you to learn from it or check how something is implemented. You could even modify the code and create your own private version of the compiler, for instance. Except for learning purposes, this does not make much sense. It is a much better idea, to post the bug report or feature suggestion as a GitHub issue and get feedback from the team. If they approve it, either you could have your code become a part of the official release, or somebody else might implement the bug fix or feature instead of you.
Not only is all the code openly available, the language is designed out in the open as well. Currently the language team at Microsoft is planning and designing new C# 7 features. They are publishing all their notes as GitHub issues and accepting comments from the community. If you want to see what is the next version of C# bringing in, or maybe even influence how the features work and what syntax they use, NOW is the time to get involved.
Conclusion:
Although Visual Studio 2015 with Roslyn compilers has been released about 6 months ago (July 2015), project Roslyn is continuing, and is more alive than it ever was. The community is taking part along with internal Microsoft teams, submitting feedback and contributing code. Improvements to .NET Compiler Platform are being done constantly and are released as part of Visual Studio updates. Next version of languages are currently in a planning and design phase. Third party developers are learning about Roslyn and finding innovative ways to use it in their own software projects. If you are interested in any of the above, do not hesitate to take a closer look at Roslyn. You just might end up liking what you discover!
This article has been editorially reviewed by Suprotim Agarwal.
C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.
We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).
Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.
Click here to Explore the Table of Contents or Download Sample Chapters!
Was this article worth reading? Share it with fellow developers too. Thanks!
Damir Arh has many years of experience with software development and maintenance; from complex enterprise software projects to modern consumer-oriented mobile applications. Although he has worked with a wide spectrum of different languages, his favorite language remains C#. In his drive towards better development processes, he is a proponent of Test-driven development, Continuous Integration, and Continuous Deployment. He shares his knowledge by speaking at local user groups and conferences, blogging, and writing articles. He is an awarded Microsoft MVP for .NET since 2012.