Smooth Visualization of Big Data Structures

Find out how to efficiently visualize large network diagrams

Internet technologies like social networks, knowledge graphs, and other big data applications are on the rise. In addition to storing and processing the data, it is often necessary to visualize this data as a diagram that humans can view and analyze. However, this can be a challenging task, since visualizing large data sets tends to pose various challenges.

Reduce the Displayed Data: Use a Drill-Down Approach

First of all, it should be evaluated whether it makes sense to display a vast amount of diagram elements at the same time. In many cases, the data can be reduced to a relevant subset, which improves visual cluttering and makes the relevant data easier to understand. Displaying all the data at the same time in contrast often leads to visual clutter, and a typical “hairball” graph that might look impressive, but does not provide much insight to the viewer.

Typical Hairball Network

There are several approaches to reduce the data, like filtering, drill-down, merging of elements, or a combination of those. Which strategy fits a use case best depends on the nature of the data and the goal of the visualization, i.e., what aspects of the data should be pointed out.

Grouping and Folding

In some cases, however, displaying a large number of elements at the same time is unavoidable. If users have not considered this necessity when designing the code that generates the diagram visualization, it can lead to serious performance issues. However, there exist several optimization strategies when processing the diagram data, which can prevent such performance problems.

How to Optimize Visualization Performance

The root problem of visualizing large data sets is the rendering of a sheer amount of elements. It might still be feasible to generate a static picture, but an interactive scenario, where items can be added, modified, and removed, is more challenging. The visualization has to be updated each time the data changes, while it still has to offer acceptable performance, i.e., high frame rate and responsibility.

There exist a few general optimizations that can apply to almost any case and some that apply only to specific scenarios. In general, the best-suited strategy depends on the exact use case.

Choose a Fast Enough Rendering Technology

Depending on the platform, there might exist several visualization technologies that offer high rendering performance. Some might produce a higher-quality rendering with features like vector graphics, anti-aliasing, others might provide better performance with concessions to rendering quality. For large diagrams, users have to choose a suitable rendering technology that provides reasonable performance and produces an acceptable quality at the same time.

For example, modern browsers provide several ways for web applications to visualize data. SVG is a technology that produces high-quality, scalable, and printable renderings. However, due to the amount of DOM objects, SVG can easily lead to performance issues when displaying a large number of elements, especially if they have to be updated frequently. In this case, switching to canvas rendering, which draws all elements into one canvas DOM element, can be a better option since it can significantly improve performance. The drawback is that in this case, users lose the scalable vector graphics of SVG. Another option is using WebGL rendering that utilizes the PC’s graphics processor. Similar technologies are available for other platforms as well. In general, it’s a tradeoff between choosing the technology and settings that are good enough in performance and still provide a reasonable graphic quality.

Carefully Choose Diagram Item Visualization

When rendering many elements, the visualization of each item must not be too complicated. For example, a drawing that performs perfectly fine with 100 nodes can lead to issues in the case of 1000 nodes. Usually, drawing a simple rectangle performs better than using a complex visualization with a lot of rendered objects. Thus, users have to make sure that they don’t use high-resolution images where it is not necessary. Especially text rendering can easily lead to performance issues. So users have to find a balance between appealing item visualizations that convey all the relevant information and simple enough drawings to ensure satisfying performance.

Reduce Visual Complexity With Level of Detail Rendering

Level of Detail rendering is a great approach to improve visualization performance without losing any details. It allows using complex item visualizations while at the same time provides high performance.

This technique applies to diagram applications where the user can zoom the diagram. The idea is to render the item details only when zoomed-in above a certain threshold and switch to a simpler visualization when zoomed-out. When zoomed-in, fewer items are visible on the screen and have to be rendered at the same time, thus making it possible to use a more complex visualization without affecting performance too much. When zoomed-out, more items are visible and have to be rendered, making it necessary to switch to a plain visualization.

Usually, there is a zoom level below which the text on diagram items is not readable anymore because it gets too small. In this case, the text can safely be hidden without losing any information. Once the user zooms-in above the threshold, the text becomes visible again. This approach can be applied to any visual elements to reduce the complexity of the drawings and improve performance.

Level of Detail can also be applied to the rendering technology mentioned in the previous section. For instance, a web application can switch from high-quality but slow SVG rendering to high-performance, but visually simpler WebGL rendering when the user has zoomed-out below the threshold. The visualization can benefit from both technologies, i.e., high-quality rendering when zooming-in on details and high-performance when zoomed-out to get the overview.

Only Draw What’s Needed

A diagram application should only render the parts of the diagram that are currently visible on the screen. The items that are outside of the view box should not be rendered at all, which is usually already the default behavior if someone uses a high quality diagramming tool or library.

Also, when using a visualization technology that works with objects, like WPF, JavaFX UI controls, SVG, etc., the rendering code should make sure that it updates only the necessary parts of the visualization at each render cycle. In the best base, only the recently modified properties are updated to the new values so that the rendering engine can use its caching mechanisms to a maximum. In the worst case, the whole object tree representing the diagram is re-created at each render cycle, which usually results in a significant performance decline.

Using Static Drawings

For some applications, where the underlying data does not change too often, it can be a viable solution to use a static drawing (either vector or pixel image) to display the diagram. This static image is created once at the beginning and can be updated when the data changes. This approach results in a high performance since only a single image has to be rendered. Changes to the data might be displayed with a latency since the whole image has to be re-created.

Use a Diagramming Tool That Provides Performance Optimizations

yFiles is a commercial programming library explicitly designed for diagram visualization. It supports all of the performance optimization strategies described in this article. Due to its generic nature, it gives users the freedom to visualize any connected data. Also, it provides features that help to reduce a large diagram to the relevant parts, like grouping, folding, and filtering.

Large Graphs Demo

A Large Graphs Sample Application is part of yFiles for all platforms except WinForms. This example application allows interactive exploration of standard approaches to performance tweaking. The available settings are platform-specific.

Performance Optimization HTML
Performance Optimization Java
Performance Optimization WPF

Optimizing Layout Calculation

Automatic layout algorithms calculate suitable locations for the graph elements and, thus, help to gain insight into the visualized data and reveal information about the underlying structure in a fast and easy way.

Note that there are many different types of layout algorithms with different capabilities and drawing styles. Of course, in general, the runtime increases with the graph size. However, while some of the algorithms are quite fast, even for large graphs, others are quite slow and only suitable for diagrams with up to a few hundred elements.

Most of the various layout algorithms provided by yFiles support the specification of a preferred time limit called maximum duration. Dependent on the specified time as well as graph size, the layout algorithm can automatically adjust its internal settings and skip/disable some of the optimization steps. This option significantly reduces the runtime for large graphs even though it may still exceed the maximum duration since the layout algorithm always has to find a valid solution. Note that, in general, quality and runtime are conflicting objectives and, thus, limiting the runtime usually reduces the layout quality. While for smaller graphs, quality is often the main objective, the required runtime gets more critical for larger ones where the primary focus does not lie in showing every detail.

Run the Layout Styles Showcase Application that comes with yFiles to see all of yFiles’ layout algorithms in action.

Examples and Source Code

On all platforms except Windows Forms, yFiles comes with the Large Graphs Sample Application that demonstrates standard performance optimization strategies. The Layout Styles Showcase Application is available for all yFiles platforms.

The source code of these sample applications is part of these yFiles packages and available on the yWorks Github repository:

Why yFiles?

Most complete solution

Since 2000, yWorks is dedicated to the creation of professional graph and diagramming software libraries. yWorks enables clients to realize even the most sophisticated visualization requirements to help them gain insights into their connected data. The yFiles family of software programming libraries is the most advanced and complete solution available on the market, supporting the broadest range of platforms, integrations, input methods, data sources, backends, IDEs, and programming languages.

Perfect match for all use-cases

yFiles not only lets you create your own customized applications but integrates well with your existing solutions and dashboards on the desktop, on mobile, and on the web. Developers can use concise, rich, complete APIs to create fresh, new applications and user experiences that match your corporate identity and exactly fit your specific use-cases. Browse and choose from hundreds of source code demos and integrations to get ideas and get started in no time.

Honest, simple licensing

yFiles enables white-label integrations into your applications, with royalty-free and perpetual licensing. There are no third party code dependencies.

Industry-leading automatic layouts

yFiles has got you covered with a complete set of fully configurable, extensible automatic layout algorithms, that not merely render the elements on the screen but help users understand their data and the relationships just by looking at the diagrams.

Unmatched customizability

Decades of work went into the creation of the most flexible, extensible, and easy to use diagramming APIs that are available on the market. Everything may be customized with yFiles: data acquisition and import, graph creation, display, interaction, animation, layout, export, printing, and third party service connectivity.

Algorithms included

With yFiles, you can analyze your graphs, connected data, and networks both on the fly and interactively with a complete set of efficient graph algorithm implementations. Calculate centrality measures, perform automatic clustering, calculate flows, run reachability algorithms, find paths, cycles, and dependencies. For the best user experience, use the results to drive the visualization, interactivity, and layout.

Unequaled developer productivity

Developers quickly create sophisticated diagramming applications with yFiles. The extensive API has been carefully designed and thoroughly documented. There are developers’ guides, source code tutorials, getting started videos, and fully documented source code demo applications, that help to realize even the most advanced features. Inline API documentation lookup for all major IDEs with hundreds of code snippets and linked related topics make writing robust code a breeze. Integration samples for many major third party systems help in getting productive, quickly.

Not just a static viewer

With yFiles, you can do more than just analyze and view your data. Create interactive, deeply integrated apps that don’t just let you consume data sources, but also enable users to create, modify, and work with both existing and changing data. Integrate with third party services to automatically trigger actions and apply updates. With yFiles, there are no limits: you decide what your app can do.

High-performance implementations

While it is recommended not to overwhelm the end-user with overly complex graph visualizations, of course, all aspects of the library have been prepared to work with large amounts of data. Developers can create both high-quality diagram visualizations and rich user-interactions, as well as configure algorithms and visualizations to perform great for even the largest graphs and networks.

Generic data acquisition

You don’t need to let your users create the diagrams from scratch or use a particular file format. yFiles enables you to import graphs from any data source which is accessible via an API. Programmatically build the in-memory model using an intuitive, powerful API. Update the diagram live in response to external events and changes.

World-class support

Get the best support for your development teams. Directly connect with more than a dozen core yFiles library developers to get answers to your questions. If you don’t have the time to do the implementation or your team is not large enough to do the implementation, let yWorks help you with consultancy and project work to get your team and apps up running, quickly.

Proven solution

Customers from all industries all over the world have been using yFiles for almost twenty years for both internal and customer-facing applications and tools. See the references for a non-conclusive list.

Frequently Asked Questions

What is yFiles?

yFiles is a software library that supports visualizing, editing, and analyzing graphs and graph-like diagrams. It is not a ready-to-use application or graph editor. Instead, it provides a component for graph visualization, graph editor features, and an extensive set of algorithms for automatic data arrangement and graph analysis. Software developers can use yFiles to display, edit, and analyze diagrams in their own applications. yFiles is available for many platforms.

Which platforms does yFiles support?

Right now, yFiles supports HTML / JavaScript, Java (Swing), JavaFX, .NET (WinForms), and WPF.

What kind of applications can I create with yFiles?

Developers can use concise, rich, complete APIs to create fresh, new applications, and user-experiences that match your corporate identity and exactly fit your specific use-cases. yFiles enables white-label integrations into your applications, with royalty-free and perpetual licensing. Any application that works with or displays relational data in the form of graphs, diagrams, and networks can be built with the help of yFiles.

What devices can I target with yFiles?

yFiles not only lets you create your own customized applications but integrates well with your existing solutions and dashboards on the desktop, mobile, and the web. There are versions of yFiles available for all major platforms and frameworks.

How extensive is the graph API of yFiles?

yFiles offers the most extensive graph layout, visualization, and analysis APIs available commercially. In total, there are around ten thousand public API members (classes, properties, methods, interfaces, enumerations). yFiles uses a clean, consistent, mostly object-oriented architecture that enables users to customize and (re-) use the available functionality to a great extent. API components can be (re-)combined, extended, configured, reused, and modified to a very high degree. It is not mandatory to know the complete API, of course. Most applications only require a minimal subset of the full functionality, and the advanced functionality and APIs may only be required for implementing unique requirements.

As a developer, what can I expect from yFiles?

yFiles helps developers quickly create sophisticated diagramming applications. The extensive API has been carefully designed and thoroughly documented. There are developers’ guides, source code tutorials, and fully documented complete source code demo applications that help to realize even the most advanced features. Inline API documentation lookup for all major IDEs with hundreds of code snippets and linked related topics help in writing robust code, efficiently. Integration samples for many major third party systems help in getting productive, quickly.

Is yFiles Free?

No. yFiles is a commercial software library. If you decide to use yFiles in your application, you’ll have to pay a one-time fee. You also have the option to subscribe annually for technical support and updates.

How does the licensing work for yFiles?

yFiles enables white-label integrations into your applications, with royalty-free and perpetual licensing. There are no third party code dependencies. Licensing basically works on a per developer basis. Please refer to the pricing information and software license agreements of the respective product for more details.

What kind of support can I get for yFiles?

The yFiles libraries come with fully documented demo applications, detailed API documentation, and extensive developers’ guides. Apart from that, yWorks also offers professional support services for your development teams. They can connect directly with more than a dozen core yFiles library developers to get answers to their programming questions. Optionally, if you don’t have the time or necessary team, yWorks can help you with consultancy and project work to get you and your apps up running quickly.

How is the release cycle for yFiles?

There is no public roadmap for yFiles. yFiles usually gets a new major feature release about every 10 to 15 months, with bugfixes or minor maintenance releases in between as required. Typically there are between one and five bugfix releases for each major release, and previous releases get important bugfixes, too. yWorks tries very hard to keep the libraries and APIs backward compatible so that customers can update to the newest version of yFiles regularly with little to no effort and still benefit from performance improvements and new features.

Can I edit my graphs with yFiles?

With yFiles, you can do more than just analyze and view your data. You can have interactive, deeply integrated apps that don’t just let you consume data sources but also enable users to create from scratch, modify, and work with both existing and changing data. Integrate with third party services to automatically trigger actions and apply updates in real-time and publish changes to third party systems while the user works with the graph. It’s up to you to decide what your app can do.

What kind of layouts does yFiles support?

yFiles comes with the most extensive set of fully configurable, extensible automatic layout algorithms, that not merely render the elements on the screen but help users understand their data and the relationships just by looking at the diagrams. yFiles includes hierarchic, organic (force-directed), orthogonal, tree-like, radial, balloon-like, and special purpose layouts. yFiles also supports incremental, partial, and interactive layouts, as well as various edge routing and automatic label placement algorithms.

Are the layout algorithms configurable?

Layout algorithms support various settings and constraints and are fully customizable in code. They support different node sizes, nested groups, bundled edges, orthogonally and octilinearly routed edges, consider and automatically place node, edge, and port labels. Nodes may be partitioned and clustered, and different layout styles can be mixed in the same diagram.

What kind of graph analysis does yFiles support?

yFiles lets you analyze your graphs, connected data, and networks both on the fly and interactively with a complete set of efficient graph algorithm implementations. Choose from a range of different centrality measure implementations, automatic clustering algorithms, network flow algorithms, reachability and connectivity algorithms, pathfinding variants, cycle, and dependency analysis algorithms. For the best user experience, use the results to drive the visualization, interactivity, and layout.

What parts of yFiles can be customized?

yFiles has the most flexible, extensible, and easy to use diagramming APIs that are available commercially. Every aspect of the functionality is customizable with options ranging from high-level configuration settings, down to low-level implementation overrides: data acquisition, import, graph creation, display, interaction, animation, layout, export, printing, and third party service connectivity.

How can I get my data into yFiles?

End-users don’t need to create the diagrams from sketch or use a specific file format. yFiles lets you import graphs from any data source that is accessible via an API. Developers can populate the in-memory model using an intuitive, powerful API, directly connecting to their preferred data sources. Diagrams can be updated live in response to external events and changes.

How can I get my diagrams data back from yFiles?

The in-memory graph model lets you export all the information to any system and file format. There are built-in export options to various file and image formats, but as a developer, you can create your own glue code to connect to arbitrary data storage systems and third party services.

Is the diagram size limited?

Theoretically, the only limiting factor for the number of graph elements is the size of the computer’s memory. In practice, performance is also a limiting factor. For the vast majority of use-cases, yFiles delivers best-in-class performance out-of-the-box. For very large visualizations and data-sets, there are options available that let developers tune between features, running-time, and quality of the results. yFiles can deal with graphs of any size and is only bound by the memory available and the runtime complexity of the algorithms. Large graphs may require adjusting the default settings and performance depends on more than just the number of elements in the diagram, but also the structure of the graph, the algorithm and configuration, as well as platform and hardware capabilities.

Who is using yFiles, already?

Customers from almost all industries all over the planet have been using yFiles for nearly twenty years, to create both internal and customer-facing applications and tools. Clients include both single developers and the largest corporations and organizations in all of academia, public and governmental services, and of course, the commercial space. See the references for a non-conclusive list. Naturally, there are the big well-known software corporations among yWorks’ customers (unfortunately only some of them allow yWorks to list them on the references page), but there’s also a great lot of companies that are not traditionally known for software, but who still have their own IT departments create software for their intranet or customer-facing applications. And last but not least, smaller companies without IT departments that let third party implementors create useful diagramming applications with the help of yFiles for them. yFiles at its core is a generic diagramming component that is use-case agnostic and can be used to create graph and diagramming-centric applications for any business domain that requires working with or displaying connected data.

How long did it take to implement yFiles?

yFiles started as a university project at the University of Tübingen in the late 1990s. Since 2000, yWorks has taken over all development and has been working continuously with a core layout-team of two to eight developers on improving the layout algorithms. The layout algorithms alone, as of 2019, took more than sixty development years to implement. A team of more than 20 developers has been working on the implementation for the visualization and interaction and the support for the various platforms yFiles supports, totaling in more than a hundred years of development for the visualization. Porting yFiles to a new platform in the past took between three and about 15 development years. Most platform variations were implemented in between six and ten calendar months.

How long has yFiles been around?

yFiles started as a university project at the University of Tübingen in the late 1990s. The company yWorks was founded as a spin-off of the university in 2000 when the first commercial customers wanted a license for yFiles. Since then, it has been developing and improving the library. It all started as a Java library, and over time, yWorks improved and even rewrote large parts of the library to add new features and support new platforms.

Who is the company behind yFiles?

yWorks is the company behind yFiles. It was founded as a spin-off of the University of Tübingen in the year 2000 specifically for licensing and supporting yFiles commercially. The German company is a privately-held, headquartered in Tübingen. More than 25 employees are working at yWorks, about 20 of which are developers, working on yFiles and the tooling around the libraries. The library developers also provide support and implementation services to yFiles customers. So as a developer, you will get first-class, highest level support directly from the team that implements the libraries.

What does yWorks specialize in?

Since 2000, yWorks is dedicated to the creation of professional graph and diagramming software libraries. The software yWorks creates, enables customers to realize even the most sophisticated visualization requirements to help them gain insights into their connected data. Their main product is the software programming library family yFiles, which is the most sophisticated and complete solution available for diagramming applications on the market, supporting the broadest range of platforms, integrations, input methods, data sources, backends, IDEs, and programming languages. yWorks has set a track-record in providing the most extensive layout and diagramming solutions for developers on all major platforms. In addition to creating, maintaining and supporting the libraries, yWorks also provides professional consultancy services in the area of visualization and diagramming. In addition to that, yWorks also provides a set of smaller software tools, both free and commercial, end-user facing and for software developers, closed-source and open-source.

Does yWorks own all the intellectual property for yFiles?

yFiles does not depend on any third party library, except of course at runtime, where it depends on the runtime of the platform. yWorks owns the IP for all implementations in the core yFiles library. Some demos show the integration and make use of third party software, but they are not required for other cases.

Which papers and algorithms does yFiles implement?

The list of algorithms implemented by yFiles is long. For the common graph algorithms, we use the traditional implementations with the standard optimizations. For many of the layout algorithms, ideas for the implementation base on publicly available papers. Some algorithms (specifically the orthogonal layout and the balloon layout) we created and helped with the creation of the algorithms and (co-)published the papers for the algorithms. Most layout algorithms have been vastly modified, tuned, and enhanced, though, and don’t follow the original implementation ideas, anymore. yWorks added useful features to these implementations to make the algorithms work in less theoretical environments. We removed previously existing constraints of the original implementations and added new ideas to make the algorithms useful for real-world usage. For most of these changes and improvements, no papers have been published.

Can I get the papers for the layout algorithms used in yFiles?

For some of the algorithms, you will find papers that describe the core idea of the layout algorithms. For most algorithms, yWorks massively enhanced and modified the algorithms to support more advanced features that are frequently required in real-world diagrams. For these modifications, we did not publish any papers. As a commercial yFiles customer, you can obtain a license to the source code of yFiles where you can read, learn about, and modify the algorithms in documented source code form, according to the license terms.

Hide Description
Open in yEd LiveDownload