Posts labeled "From the labs"

Recognizing graphs from images

Projects From the labs yFiles WPF OpenCV

Popular Blog Posts

Once a year the yWorks office is alive with a very special atmosphere: The project days. Small groups of developers gather together to work on projects that are not necessarily connected to the everyday work or the core products of the company. This year, four brave developers set out to new lands and tried to parse photos and screenshots of diagrams as actual yFiles graphs without ever having touched computer vision topics before. This is their story. Motivation Do you remember the last time you've created a diagram on paper and found yourself re-creating it in a general diagram editor like yEd? This use case isn't that uncommon, because we often sketch out our ideas in form of diagrams during meetings or plan certain processes on the whiteboard. Usually, we want to digitize them afterwards in order to make small incremental changes and insertions or even apply an automatic layout algorithm to clean up the chaotic arrangement. Slightly easier input images are programmatically generated diagrams, e.g. printed in literature, that have consistent styling across all items. Still it would be great if we could create a digital graph model from this image. In this project, we try to make a first approach towards this general problem and see whether we can acquire new experiences in the field of computer vision. Hide DescriptionOpen in yEd LiveDownloadGeneral Approach After some research on the topic, we decided to take a shot at it with Python and OpenCV for parsing the image and eventually create JSON output describing the relevant diagram elements with layout and styling information. This output could be easily consumed by any general purpose diagramming library, though in our case, we will build a yFiles WPF application for rendering the parsed diagram. Processing Pipeline We decided on a processing pipeline which should give us the flexibility to simultaneously work on separate parts of the project. The first and basic step is the image segmentation where the image is broken down into disjoint regions of the overall image, each of which contains either a node, or an edge. Those regions can then be further processed independently to detect different features (e.g. color, shape, edge bends, etc.). Of course, the quality of the segmentation step impacts all the subsequent processing on the separate regions. Segmentation Given the entire input image, we first need to separate it into node and edge regions, which simplifies the identification of more features in the resulting regions. All subsequent operations are performed based on this initial segmentation which is crucial for the overall result, since elements that haven't been segmented properly will be missing or merged in subsequent steps. We've explored several approaches here and found that there doesn't seem to be one single approach that works for arbitrary input. Therefore, we kept a selection of segmentation algorithms and chose one of them depending on the current input. The following sections describe the segmentation algorithms we've implemented in more detail. This list is not exhaustive, as we had several ideas for other algorithms that could work better for certain inputs. Automatic Color-based Segmentation This segmentation approach tries to distinguish nodes, edges and background based on their color with the requirement that the elements of each category have the same color (which is unique for each category) and with the restriction that we only have three unique colors in the image. At first, a 3-color quantization step is applied to the image to reduce errors from anti-aliasing or uneven exposure. From the resulting colors, we assume that the most frequent color marks the background, while the second-most frequent color represents the nodes and the least frequent color is the edge color. Although this sounds like a pretty strong assumption, it works surprisingly well for many programmatically-generated graphs and also for certain hand-drawn or scanned pictures that share the same traits. If the node and edge colors are too close, anti-aliasing sometimes causes artifacts near the border of nodes and edges that are wrongly classified. Such artifacts can later be removed due to their implausible size. However, the obvious drawbacks are that it doesn't work for images where nodes may have different colors, or are not filled. The segmentation also fails if the ratio between background, node, and edge pixels is not as expected, e.g. if there are many edges, and rather small nodes, the edge color will be incorrectly assumed to be the node color based on its frequency in the image. Morphology-based Segmentation The second approach to the segmentation works with morphological transformations on the input image. Morphological transformations change a binary (i.e. black and white) image globally according to certain rules. The first step for this is to convert the input image to grayscale and subsequently to black and white to be able to use morphological transformations. Then, we find the outermost contour in the image and assume this to be the foreground (i.e., the graph). This contour is used as the initial mask. We then try to remove edges in the image by running multiple erode passes on this image. Erosion is an operation that removes pixels from the outside edge of shapes. This implies that thin features, like edges, eventually disappear after multiple erode steps, while nodes will get smaller, but still remain in the image. After all edges have disappeared, we reverse the process with dilation, which adds pixels on the outside edge of shapes, to ensure that the nodes are roughly as large as they were in the beginning. We can then run OpenCV's findContours() on the image to detect our nodes. Given the nodes' contours, it is trivial to create an edge mask in which we can detect the edge regions for our segmentation. A major challenge in this approach is to detect how often we need to erode the image until the edges are removed. The number of erosion steps depends on the thickness of edges, and we employed a heuristic where we observe the number of contours in the image for each step. At some point, the image will disconnect into several new contours and multiple subsequent erosions won't significantly change the overall number of contours. This is assumed to be the point where the edges disappeared and the nodes on both ends of the edge now form their own individual contours. This heuristic has been introduced to be able to segment images where the edge thickness is similar throughout the image, but can vary from one image to the next, without having to adjust constants in the code. This segmentation approach works well for diagrams where the color-based segmentation is not suitable, for example when nodes and edges have the same color. However, it fails for diagrams where the edge thickness differs strongly due to the adaptive cut-off for the erosion steps. In those cases, certain regions of the diagram won't disconnect and, therefore, be considered as an entire node region, which causes problems for the node detection in this area. Unfilled nodes with just an outline present a challenge because we cannot separate foreground and background globally by color. Instead we must use the surrounding contour, which requires that edges in the graph cannot enclose a background region. This would then be detected as a node instead Manual Marking of Nodes We had a fair number of test images we wanted to use — especially for later stages of the pipeline — that couldn't be segmented by either of the other two approaches. We added a fairly simple segmentation approach where the user marks each node by clicking on it. Nodes are then detected by taking the color at that point and finding the whole region around it. After all nodes have been marked, the image is converted to black and white, nodes are removed from that image, and the remaining contours are detected as edges. This works well for graphs where the nodes have many different colors (negating the automatic approach), or regions enclosed by edges (preventing the morphological approach from working). An obvious drawback is that it's tedious when there are many nodes. Furthermore, edges cannot have the same color as their adjacent nodes, because otherwise the edge would be detected as part of the node. Node Labels Besides the general segmentation in edges and nodes, we also detect label regions within nodes. Given the already detected node regions, label detection is rather simple, because running OpenCV's findContours() on a threshold image of the node region provides not only the outline contour, but also all nested contours. Any nested contour marks a character inside the node. The bounding box of these nested contours then builds the label region. There are several problematic aspects to this approach. First, the label needs to be enclosed by the node, otherwise the label will be clipped at the node's bounds. Also, the font to node fill color contrast needs to be large enough so that it does not get lost in the image threshold and the node needs to have a solid fill and no empty holes that may be detected as inner contour. Although these are quite some limitations, there are still many use cases that fulfill the requirements with which the labels can be detected properly. In the resulting graph model, the labels can then be either used as clips from the original image or an actual text label can be created by applying OCR. Node Detection Based on the image clip that is defined by the node region provided by the segmentation, we try to further identify properties for the nodes. The goal is to detect the main features that are characteristic for a node, i.e. the shape (rectangle, triangle, ellipse, ...) and the color, where we want to detect the fill color, the outline stroke as well as the stroke thickness. Node Shape Detection In this step, we want to determine the shape of the each given node region. The pre-processing in the pipeline already provides an outline of the contour in the node region, so the challenge in this step is to decide whether the given contour is a rectangle, triangle, ellipse etc. The contours are not normalized in any way, therefore we need to consider the different sizes as well as to determine whether they are rotated or not, because we also want to detect, for example, whether a triangle is upside down or not. In general, OpenCV comes with a matchShapes() function with which a similarity score of two given contours can be calculated. This function is scale and translation invariant. However, it is also rotation invariant which is why we decided to implement another approach for this task. What turned out to work quite good is creating specific shape masks with the same size for each separate node region and shape that we want to detect. With these masks, we calculate the difference between the actual contour mask given by the segmentation and these shape masks. The similarity then correlates with the non-zero entries after bitwise XOR combination of the inverted shape mask and the contour mask of the node itself. The more non-zero entries this bitwise XOR combination yields, the better the reference mask matches the original contour. Although this approach requires to re-create all reference masks for each node, the performance still turned out to be good. The mask creation and bitwise matching are cheap operations and at least as fast as using the built-in matchShapes() method. Fill and Stroke Detection Similar to the shape detection, the color is one of the most important properties of a node when creating a graph from an image. To detect the fill color of the node, we decided to use a k-means clustering with k=10 on the color of the pixels within the node contour but ignoring the label region if there is any. The cluster with the largest number of pixels determines the main fill color. This simple approach turned out to work quite well as long as the node region does not contain a large number of anti-aliasing pixels. However, for small ellipses it often produced a mixed color depending on the ellipse's stroke because of the anti-aliasing pixels on its outline. Now that we have our k-means clustering for the color, the most obvious approach for the stroke color would be to just use the second largest cluster that is not the background color. But as it turns out, this only works for very specific shapes and sizes and thus often yields wrong results. Our approach for detecting the stroke color is to generate one pixel slices from the contour of the node. We iterate from the outside to the inside and create an even tighter slice for each iteration. During the iteration, the main color for each slice is picked using k-means clustering. Once this color is similar enough to the fill color of the node, we stop the iteration and know that the pixels of all slices build the stroke outline of the node. Therefore, one last k-means color clustering can be applied to those pixels in order to get the stroke color. This approach can also be applied inside-out when the given node contour does not contain the stroke outline, which may be the case depending on the initial segmentation approach. Another benefit of this iterative approach is that it implicitly yields the stroke thickness as well, since it is the iteration count for generating the one pixel slices until the stop condition is met. Edge Detection Of course, a graph usually does not consist only of nodes but of edges, too. We already get regions for each individual edge from the segmentation. We now need to figure out where an edge starts and ends, as well as how its path looks like. Later on, we also try to detect visual features so that the resulting edge looks as close to the original image as possible. These include the exact positions where the edge attaches to the node (the port positions), the edge thickness, color and direction. Edge Geometry To start with, we need to convert the edge region, which encompasses the whole edge path, to something that is easier to work with. To this end, we perform skeletonization by morphological thinning. This skeleton retains the basic shape of the original input and also stays connected, just reduced to an one-pixel wide line. This makes it easy to detect certain features, such as end points of a line, corners, or crossings, simply by looking at a pixel and its immediate neighborhood. One remaining problem is that - especially for thicker edges and arrowheads - there are a lot of branches, or spurs, leading off from the core skeleton. We prune those away by successively identifying end points via a hit-and-miss transform and removing them, until only two end points remain. Those two end points are then matched to their closest node regions to determine the two end nodes. Furthermore, since we have two end points and we know that the whole line is connected, we can just walk along the neighboring pixels from one end point to the other to obtain the edge path. Of course, this path has a bend point at each and every pixel, so we apply the Douglas-Peucker algorithm to approximate a polygonal line that better matches the original image. We also need to figure out the port positions, where the edge attaches to the node. Due to the pruning step that we applied earlier to remove the spurs and retain only two end points, we've also pruned away the actual end points of the line, which are now further away from the node. This isn't much of a problem for determining the end nodes, but it means that we've lost part of the edge path near the node and cannot easily determine the actual position on which it was attached at the node. To remedy this, we've started with just the two end points from earlier and used morphological dilation to grow them until we find the intersection with the nodes. This is somewhat error-prone and not ideal, and more attributed to our inexperience and and the fact that we didn't have a better idea for the earlier stages as well. More on that below when we compare our approach to existing approaches (which we've found after settling on this approach). Edge Styling To further capture the fidelity of the detected edges, we also detected two main features to make them look more similar to their original counterparts from the input: Color and thickness. The color is determined by taking the dominant color along the edge's skeleton. This skeleton is completely inside the detected edge region and, thus, only yields pixels belonging to the edge. This works well for many edges, but can result in the wrong color when the edge is very thin and anti-aliased, since the algorithm picks up more anti-aliasing pixels than pixels of the actual edge color. Thickness is determined by taking the number of steps required for the skeletonization in the beginning. The process of obtaining the skeleton takes away pixels from the outside of the shape until nothing else can be removed without breaking the shape, so the number of those steps is roughly half the thickness of the edge in pixels. Due to some inaccuracies on how we've implemented skeletonization this doesn't fit exactly, so a better factor has been determined empirically. We've also included a heuristic to detect whether the original edge path was curved and apply a smoothing step to the edge style on the yFiles side to mimic the original curvature. Directed graphs have a direction for each edge, usually signified by an arrowhead on the target end of the edge. To detect those arrowheads we start with the original skeleton, where arrowheads are represented by two long spurs near the target end of the edge. We then prune away the shorter spurs again and see whether we end up with a cluster of exactly three end points near one end of the path. This is then interpreted as the arrowhead and defines the edge direction. Showcase In this section, we want to show some of the results for different input images. To test our different approaches, we built a small C# application where we could experiment with the algorithms on different inputs. The application itself is quite barebone and provides three separate panes. In the top left pane, the input image is displayed. The bottom left pane shows the segmentation regions that are overlaid over the original image. The larger right pane is an actual yFiles WPF component where the the parsed graph is created and displayed, meaning that all the items that are displayed in this pane are graph model items. Click the image to open the album. State of the Art During our work on the project we encountered two existing publications tackling the same problem: Optical Graph Recognition by C. Auer et al. (2013), and Optical Edge Detection by R. Opmanis (2018), which improves edge detection compared to the earlier paper. Surprisingly for us, our approach was fairly similar to what the actual professionals used, even though we had very little prior experience in Computer Vision. Major differences between the existing publications and our approach were the scope of the work and the areas on which we decided to invest time and effort. Mainly, the existing approaches concentrated on getting the graph structure correct. Our approach, being more visually driven, aimed at improving the fidelity of the recognized graphs, i.e., shapes, colors, labels, and edge directions. Those are topics often hinted at in the publications as being useful to pursue further. Those differences naturally result in different features and drawbacks for each of the approaches. The 2013 paper by C. Auer et al. employs a morphological segmentation approach that requires images of graphs with filled nodes and edges to be significantly thinner than the node size. Edge detection is handled by also identifying crossings and in many cases correctly recovering the original path of all edges, including those that cross each other. Their approach to edge detection also recovers port locations faithfully and thus, detecting the wrong source or target node for an edge is next to impossible. The 2018 paper by R. Opmanis uses a different approach to edge detection and improves detection of crossing edges as well as allowing dashed edges. Our approach did not attempt to handle edge crossings, instead we've invested more time in different segmentation approaches, which allows to lift some of the restrictions of the earlier approaches. We've also conflated the segmentation step with the pre-processing step, which enables segmentation approaches based on color. Under certain conditions we can correctly segment images with outlined (instead of filled) nodes, and our edge detection can also handle wildly varying edge thicknesses. However, both of those improvements place additional constraints upon either the colors or the graph structure of the input. Our edge detection, as mentioned, is less sophisticated and sometimes has problems with identifying the right source or target node as well as the exact port locations. This is mostly due to us not re-evaluating our choice of algorithm and finding the mentioned papers rather late in our development process. On the other hand, we are able to detect edge color, thickness, and direction via arrowheads fairly reliably in many cases. For nodes, we also have approaches for detecting the node shape, color, and outline. Color detection for both fill and strokes works well in many cases, but has proven to be unreliable with very thin features and anti-aliasing, where the apparent color to a viewer is not the one we can recover. Concluding, both the 2013 paper and our approach share a number of similarities, mainly in how morphological transformations are applied. By combining the existing approaches with our improvements to fidelity as well as with what we learned, we're sure the overall result in optical graph recognition can be further improved. Conclusion and Future Work We believe that our approach can be merged well with existing approaches to improve several aspects on both sides. The mentioned existing approaches generally have better recognition of the graph structure, including edge crossings. In contrast, our approach offers better fidelity that results in graphs that look very much like the input, including labels, shapes, and colors. For the task of digitizing a graph from an image and working further with it, styling information is very nice to have as part of the conversion process instead of having to add it manually afterwards. Of course, our own algorithms are very much at a prototype stage right now — due to limited time — and could certainly use improvements in many areas. Many stylistic aspects we detect are fraught with edge cases and smaller problems: edge thickness can be unreliable to detect, colors of thin edges or outlines can be subtly wrong due to anti-aliasing, shape detection can only detect a very limited set of shapes right now, etc. Perhaps the part that could use most improvement would be the segmentation, where small improvements could already result in vastly better segmentation. The hard-coded three-color quantization for the automatic color-based segmentation is an obvious limitation that can be removed to allow for graphs that have multiple node colors. Manually aiding the segmentation by either preparing a good background/foreground mask or correcting segmentation errors are also ideas that would greatly help real-world usage of such an approach, as segmentation errors will always happen, but some of them result in graphs that are hard to correct. Especially photos pose a unique challenge in that uneven exposure can make correct segmentation difficult with our current approaches. On the other hand, there are already many cases where the current version provides very good results - see the provided gallery for examples. What do you think about the topic and our results? Let us know! Just leave us a message in the social media or using our contact form.

Case Study: Intelligent Data Provision in the Automotive and Rail Industries

VIRTUAL VEHICLEResearch CenterAutomotive and Railresearch & developmentResearch & development in the automotive and rail industries is shaped by renowned international project partners with a multitude of data from various sources and structures. In a research project, VIRTUAL VEHICLE develops a prototypical digital decision-making basis: a graph database for data collection and an application for visualizing and navigating the connected data. How Austrian researchers are rethinking automotive R&D using data visualization and yFilesNew topics like e-mobility and autonomous driving present automotive companies with new challenges, particularly in the area of research and development. For example, the cycles of model development are expected to change. These cycles currently last up to eight years and will become significantly shorter in the future, especially in terms of software and hardware. Source: VIRTUAL VEHICLETo meet these future demands, research & development is rapidly moving in the direction of virtualization and digitization – away from classical experiments on prototypes, and towards numerical methods. This evolution requires new development environments with agile processes and the appropriate methods and tools. For this reason, the Efficient Development division of VIRTUAL VEHICLE Research Center in Graz, Austria - one of the leading research centers for automotive and rail industries - focuses on a functional system view, orchestration of complex product verification, and advanced information and process management. VIRTUAL VEHICLE, one of the leading R&D centers for the automotive and rail industries, is based in Graz, Austria. Future Engineering LabSource: VIRTUAL VEHICLEThe Future Engineering Lab project team is working with renowned project partners from the automotive and rail industries to develop a prototype implementation of future data provision and consistency in engineering, along the product lifecycle. Their approach is to collect the heterogeneous data sources, and then develop aggregating and view-based access to them. By virtually merging the metadata in a graph database, the team frees itself from predefined data structures (as is common in relational database systems). As a result, they can concentrate on the relationships between the data, and integrate additional data sources as needed. The researchers develop a web application whose core is data visualization. An interactive graph makes it easy to navigate through the data and serves as a digital decision-making basis for development engineers. The data is stored in native graph format in a NEO4j database.Source: VIRTUAL VEHICLEVisualizing the dataThe requirements for the visualization solution were: potential future integration into other tools and platforms, intuitive visualization of large linked data sets, the option of expanding and collapsing data points, suitable structuring of the visualization, and a high degree of freedom in realizing the project and designing user interactions. After a lengthy evaluation of commercial as well as freely available visualization tools, the decision was made in favor of yFiles for HTML, by yWorks GmbH. Dr. Bernd Fachbach, project lead“The flexibility in designing our own application, as well as the many existing, relevant implementation examples led to our decision to use yFiles.”Source: Dr. Bernd FachbachThe developed test model serves as a prototype of future data provision and consistency in engineering, along the product lifecycle. The prototype is intended as a future-proof model: a web application that allows exploration and navigation in context-based engineering information. Initial results will be published in summer 2019. VIRTUAL VEHICLE | Kompetenzzentrum – Das Virtuelle Fahrzeug Forschungsgesellschaft mbHVIRTUAL VEHICLE is a leading international research and development center for the automotive and rail industries, located in Graz, Austria. The center focuses on the consistent virtualization of vehicle development. An essential element is the linking of numerical simulations and hardware testing, which ensures a comprehensive hardware-software system design. VIRTUAL VEHICLE’s international partner network consists of:80+ international industrial partners40+ international scientific institutions (including TU Graz, TU Berlin, Fraunhofer)VIRTUAL VEHICLE is the largest COMET-funded research center and is also active in over 30 EU projects. In addition, VIRTUAL VEHICLE offers a broad portfolio of contract research for vehicle development.Download Case StudyFurther Sources:PWC Studie Automotive Eascyhttps://www.pwc.de/de/automobilindustrie/pwc_automotive_eascy-studie.pdf

yWorks releases version 2.2 of yFiles for HTML!

With the new 2.2 release of yFiles for HTML yWorks again proves to be the leading provider of professional high-quality graph diagramming software libraries. The new release contains new features, unmatched in the industry, that will both delight existing customers and will open the product to a wider audience range.In this post I want to talk about some of the highlights that we included in the new 2.2 release. Since the focus is mostly on new features, for those looking for a more general introduction to the library and its capabilities, we suggest to take a look at the features page, instead.We always make sure that any major new release brings improvements and new features to all areas of the product: There are less visible improvements that developers will benefit from, but also end-user visible features like improvements to automatic layout, and several improvements relating to the visual and interactive API parts. Still the API has remained stable and mostly backwards compatible to older releases. Both, developers new to yFiles and existing applications can benefit from the new release.Improved and New Demos Of course, we also improved and reworked many of the demos and also added some more for you to get inspired from. Here are some of the new demos and feature updates: Improvements for Automatic Layout Of course, the 2.2 release contains some features for our most flexible and most widely used graph layout algorithm for hierarchically structured graphs. But there are also nice improvements to the generic edge routing algorithm and the generic tree layout styles. Hierarchic Layout Improvements For the very wide layouts in flat hierarchies, there is now a perfect option to produce more compact bus-like drawings for the successors and predecessors of specific elements. As can be seen in the example, such an arrangement is especially suitable for star-like substructures (i.e., a node connected to multiple nodes with degree one) which can otherwise consume a lot of space and make the layout less compact. The new layout results in better legibility for the labels and makes better use of the screen real-estate in typical use-cases. Edge Router Improvements The generic edge routing algorithm can be used in those situations where the locations of the nodes are fixed and where there is no layout algorithm that does not perform an integrated edge routing along with the arrangement of the nodes. For the edge routing algorithm, two major new features have been added to the release:Integrated Labeling Integrated edge labeling allows the algorithm to choose edge routes that consider one or more labels and their size of an edge. This ensures perfect legibility of even longer labels. If necessary, the edge will be routed using a different path so that it passes through areas that can hold the label's visualization without producing overlaps.Support for Bus Structures In addition, we added support for bus structures to the generic edge routing algorithm. The routing of such buses is like the results produced by the bus routing algorithm and combines the routes of multiple edges that form a bus. Also, the edge routing algorithm is able to optionally consider user-specified bus locations and provides more options for solving other constraints.More Tree Layout Variants Our versatile tree arrangement algorithm received a number of new arrangement strategies. The first one supports the distribution of nodes into multiple branches (a kind of vertical bus) that are placed next to each other. Of course, any orientation and distribution is supported by the implementation and the strategy can be applied to parts of the diagram, only. The second new strategy is based on the existing grid placement facility and allows to create a symmetric bus-style featuring multiple rows or columns respectively.RUN DEMOView and Interaction There are many smaller and incremental improvements for the view and interaction part in yFiles for HTML 2.2. We both increased performance in many areas of the visualization and added more customization options for the developers to choose from.More Built-in WebGL Rendering OptionsWebGL had been added to yFiles for HTML with the 2.1 release as an optional third rendering technology, next to SVG and HTML5 Canvas. The 2.2 release now provides more WebGL visualization options out of the box. For the visualization of very large networks, the overview visualization, the grid visualization, and the visualization of the default manipulation handles can now be rendered using either SVG, Canvas, or WebGL, bringing together the strengths of the three technologies in a single diagram visualization.Hide DescriptionOpen in yEd LiveDownloadAdvanced CSS Styling and AnimationFor the 2.2 release we made it significantly easier for the developer to change the default color schemes and the animations for the various controls in the diagram: Tooltips and the integrated label text editor can now open and close using sophisticated CSS3 animations with minimal effort. The coloring and styling of handles, selection, focus, and highlight indicators, as well as the selection marquee is now as easy as changing a line of CSS code.RUN DEMODeveloper Highlights With every release we improve the productivity of the developers working with our software library. This means not only adjusting to new trends in the web development world, but also improving the way our APIs work and how the huge amount of features our libraries offer can be combined by software developers to create new, fascinating end-user applications.ECMAScript Modules Loading The last release paved the way for ECMAScript module loading , and it has proved to be a blessing for modern web development and the support in web development IDEs has improved dramatically for this technology. In this release we switched the complete documentation to use the more concise syntax throughout the code, samples, tutorials, and demos. All demos and existing code snippets have been rewritten and refactored to use ES modules, and we added tools that help with the automatic migration from older module loading systems to ES module loading and the other way around.import { GraphComponent, GraphEditorInputMode, ShapeNodeStyle, HierarchicLayout, } from 'yfiles' const graphComponent = new GraphComponent('#graphComponent') graphComponent.inputMode = new GraphEditorInputMode() const graph = graphComponent.graph const blueNodeStyle = new ShapeNodeStyle({ fill: 'blue', shape: 'ellipse'}) graph.nodeDefaults.style = blueNodeStyle graphComponent.morphLayout(new HierarchicLayout()) yFiles for HTML as a Node Module With the growing popularity of CLI tools like create-react-app , Angular CLI , Vue CLI , and other webpack and rollup.js based tools, the support for node and npm modules also improved a lot with the various JavaScript and TypeScript IDEs. yFiles for HTML 2.2 can now be bundled as an npm module, making integration into a large amount of build systems as easy as writing npm install --save yfiles. Currently, yFiles for HTML is not available on any public NPM repository and instead needs to be installed once, in a private repository, first.Integrating yFiles for HTML into a build tool-chain becomes almost trivial with the node module support Setting up a new project now barely requires any configuration steps anymore and code completion, build-optimization, and source code transpilers work out of the box with both TypeScript and JavaScript projects.{ "description": "My custom yFiles for HTML App", "private": true, "name": "yfiles-diagramming-app", "version": "1.0.0", "scripts": { "production": "webpack --mode production", "dev": "webpack --mode development", "start": "webpack-dev-server --mode development --open" }, "devDependencies": { "@yworks/optimizer": "^1.0.0", "webpack": "^4.22.0", "webpack-cli": "^3.1.2", "webpack-dev-server": "^3.1.10", "babel-loader": "^8.0.5", "@babel/core": "^7.4.0", "@babel/preset-env": "^7.4.2" }, "dependencies": { "yfiles": "git+https://yguy@repository.internal/npm/yfiles.git", "@babel/polyfill": "^7.4.3" } } Starting with this release we provide a webpack plugin that automatically optimizes, obfuscates, and minifies the yFiles library in production mode. We also worked hard on making the library more resilient against overly-aggressive tree shaking implementations. The chances that a buggy optimization tool-chain now breaks the functionality have been reduced and unused yFiles modules are automatically stripped from the build. We restructured the ES-modules so that now every type is available from a top-level yfiles module, making it much easier to use the available functionality.Hundreds of New Code SamplesWe worked hard to ensure that the most important parts of our API now contain code snippets and example usage code that is accessible right from the API browser and via code completion in the IDEs. More than 600 new code snippets have been added to the documentation, totaling in more than one thousand code snippets.SHOW APIFor the new release more than 1000 code snippets and examples have been added or rewritten We also updated our YouTube Playlist for Developers, of course, to feature the newest versions of APIs and tool-chains.Enhanced Collection APIGraphs are all about data and working with the elements in the data structures has always been key to the API. All collections of elements in the yFiles data structures (lists, sets, maps, collections of nodes, edges, ports, labels, bends, adjacency lists, etc.) now support additional functionality to perform frequent tasks like grouping, filtering, sorting, mapping, reducing, flattening, projecting, combining, etc. with a concise and efficient API:const widestNodes = graphComponent.graph.nodes // group nodes by their width .groupBy(n => n.layout.width, (width, items) => ({ width, items })) // and order the groups by their size .orderByDescending(entry => entry.items.size) // consider the largest two groups only .take(2) // and take all their items .flatMap(entry => entry.items) Stream-lined Graph Algorithm APIyFiles always provided the broadest range of efficient graph analysis algorithm implementations for the web platform. But the API wasn't as easy to use as the layout API and parts of it required advanced algorithm knowledge. The new release contains a completely new facade to the existing algorithms, making configuration, execution, and result interpretation consistent and simple. The new API makes it easy for developers to leverage the power of graph analysis algorithms for their visualization, layout, interaction, and analysis purposes.// configure the shortest path algorithm const algorithm = new SingleSourceShortestPaths({ // single source source: startNode, // add edge cost mapping which returns the actual length of the edge costs: edge => edge.style.renderer .getPathGeometry(edge, edge.style) .getPath() .getLength() }) // run the algorithm: // calculate paths from startNode to all nodes in the graph const result = algorithm.run(graph) // for each end node for (const targetNode of targetNodes) { // set its distance to the startNode as label text graph.setLabelText(targetNode.labels.get(0), `${result.getPathTo(targetNode).distance}`) // and mark the edge path from start to the end node for (const edge of result.getPathTo(targetNode).edges) { graph.setStyle(edge, highlightPathStyle) } } The new API makes it easy for developers to leverage the power of graph analysis algorithms If you haven't done so, go ahead and evaluate yFiles for HTML, today! Happy diagramming!EVALUATE NOW

yFiles and Neo4j: Custom Visualization Solutions

Neo4j is great for storing and processing large amounts of connected data. For example, in web analytics, where click-paths and custom events get logged, it is easy to get that data into Neo4j. Once the data is in the graph database, complex queries may be executed that help both site administrators and site owners get a good understanding of how their users and visitors use the website or ecommerce store.Sure, the results are used to improve the predictions of recommendation engines. Gut with that knowledge, shop architects can also then optimize the structure of their sites and further optimize customer experiences.However, one aspect that is often overlooked is how the visualization of graphs helps in understanding the query results and the automatic decisions of recommendation engines, as well as in planning for optimizations of customer journeys.The Neo4j browser is an invaluable tool for experimenting with Cypher queries and refactoring the schema of your database contents. The developer tool also comes with a visualization component that displays query results in a graphical manner.The simple, low-level interface is great for developers, but less technically inclined end-users prefer less-generic, highly-customized solutions that suit the exact requirements of their specific use-cases.There are several aspects that need to be considered to get the most out of data visualization and the best end-user experience:Rich VisualizationBeing able to visualize connected items and their properties in diagrams helps a lot in understanding the data and processes. For the best user-experience rich element visualizations should be used. Many generic solutions provide a master/detail view of the graph, where mainly the structure of the visualization is presented in one part of the screen. The details behind each of the elements are seen in a separate, mostly textual detail window beside the main visualization. This quickly becomes difficult to use once the diagram is explored in more detail. Users have to manually select each of the elements one by one to find more information about the element. Exporting such a diagram for reports becomes almost impossible, because the detail information is missing from the visualization or is only available for a single element. With rich item visualizations, more information can be put into the diagram and included in the visualization for every element on the screen or later in the report. For example, in a customer journey diagram, not only the name or ID of the page could be shown in the visualization, but also numeric data like “time spent on page,” “drop-off-rate,” “average page value” and “number of visitors.” These values can be added to the visualization of each element. They don’t need to be represented as naked numbers either, but data can be implemented via color-coding, gauges, level-meters, varying sizes and so on. Of course, auxiliary textual data is displayed just as well as one or more badges that indicate a certain state of the page or event (“high-performer,” “needs-attention,” “data-anomaly”). These techniques result in drastic improvements in the user experience. Just like in Neo4j, where different node labels depict a different type of entity in the database, the visualization should make it easy for the user to tell the difference between the different kinds of elements. Not all elements need to follow the same visualization scheme. Instead, the visualization should make it easy for the user to recognize which types of elements are visualized.Sometimes there is a problem with having many details shown inside the visualization: If the number of elements in the diagram or on the screen gets large – if the user needs to zoom out of the diagram to understand the bigger picture – that additional information can become distracting and make the graph impossible to decipher. In this case, sophisticated visualization solutions can use a technique called “level of detail rendering.” Depending on the number of elements on the screen, or the zoom level, element visualizations of varying detail can be employed. Once the user zooms into the diagram, more information becomes visible. If the visualization permits, switching between those levels can be animated, resulting in smooth transitions between the various levels of detail. Arranging Elements MeaningfullyAnother aspect to diagram visualization, that’s just as important as how to render the elements on the screen, is where to place those elements. For many applications, the location of the elements in the diagram can encode additional information for the user right into the visualization. Simple solutions only try to reduce the distance between elements that are interconnected by relations. More advanced placement algorithms can encode additional information into the layout of the diagram. That information could be importance or a total ordering of elements. More important elements could be rendered on top of the drawing. Or, if there is a flow of data or goods, a common main direction of the connections in the diagram helps a lot in understanding the flow through the data.But there are more subtle things than flow and order. Alignment and different connection routing styles help just the same – not all relationships must be visualized by an arrow pointing from one element to another. Especially for hierarchically nested structures and relationships, using container visualizations or partitions in the background often are the better choice and lead to less cluttered, more enjoyable diagrams. Sometimes there isn’t a single aspect or layout that suits the application domain, and sometimes the diagrams’ contents change over time. For these scenarios, it’s important to have versatile layout algorithms, as well as smooth transitions between the various visualizations. The latter helps the users keep their mental maps of the diagram data. Watching the animations is a pleasure, too.Interaction and AnimationFinally, interactivity is king when it comes to complex visualizations. If you can give the user the option to interact with the visualization, they will not only enjoy navigating and exploring the database; they will be able to do this a lot more efficiently, too.This is not just about being able to pan the drawing area. Even for the simple use-case of exploration, things like mouse-hovers, highlights, tool-tips and context menus, contextual filters, guided view port animations, etc. can improve the user-experience dramatically. A Perfect Match: Neo4j and yFilesNeo4j is the right tool to store and query your data, and the built-in browser is great for developers. For end-user facing applications, more elaborate visualization solutions should be used. For connected data, the graph visualization library yFiles is a perfect match, since it provides all the possibilities described above.Here’s how yFiles is employed in the above scenarios:With yFiles, you don’t need to just render the raw data in the database. With software-defined mappings you can show more useful abstractions of the data stored in Neo4j.Create perfect item visualizations, specific to the data that will be displayed on the screen. Level of detail rendering and custom reactive displays of the data result in easy-to-understand-and-follow diagrams and visualizations.With the right layout, the structure of the data and their dependencies and relationships can be highlighted. Clever edge-routing and labeling algorithms ensure that no information is hidden behind other items. This removes the need for the user to manually untangle the diagram. The yFiles library comes with the most complete set of diverse automatic layout algorithms for you to play with and choose from.Interactions and built-in animations help the user understand and navigate the diagram more easily and increase the user-experience.You can add user interactions and the ability to edit both the structure and the properties of the diagram. Let interactions trigger updates in the Neo4j database or in external systems.Embed the visualizations into new or existing end-user applications as a white-label solution. Create standalone applications or integrate the functionality into larger dashboards and tools. It is important to understand that yFiles is much more than an off-the-shelf application with only limited customization options. yFiles is a powerful tool for developers to create applications that meet even the most advanced requirements. Explore the demos and download yFiles today and enjoy the endless possibilities you get when combining it with the power of graph databases like Neo4j. Happy diagramming!

‘Game of Thrones’ Relationship Graph

Every year, the Graph Drawing conference organizes a contest which sets the task to visualize predetermined graph data. This year, the data consists of a graph that presents the relationships between 84 characters from the popular TV series ‘Game of Thrones’. Since our yFiles diagramming library is designed to arrange and visualize graph structures, we decided to participate in the contest. The image above shows a preview of our contribution. Read on if you want to find out how we created it or skip to the final result.Getting to know the dataFirst, we used yFiles to quickly parse the provided GraphML file into a graph in which each node represents a character and each edge between two characters indicates a specific relationship.Hide DescriptionOpen in yEd LiveDownloadApplying different yFiles layout algorithms to the graph provided insight into the overall structure of the connections. However, the data contains more information which is not depicted in these standard views:The full name of each character, its affiliation to a house, family, or group, and whether they are alive or deceased.The type of relationship; this can be a killing, an allegiance, or a family tie. In the latter case, the type further differentiates between spouses, parents, siblings, and lovers.In addition some parts of the graph are tightly connected which surely makes it difficult to follow the paths of the connections.Introducing structureSo how do we restructure the data such that a more structured arrangement can be achieved?Since the story of the TV series is all about different families and groups which are competing for power, we decided to divide the characters by their affiliation to the different houses/families.We introduced new nodes that represent the houses and connected every member of a house to the respective house node. Allegiances and family relationships between members of two different houses are redirected to the house nodes to keep them from spanning across the whole graph. Furthermore, we sized the nodes according to the number of allegiance edges to which they were incident initially. This makes more important characters easily recognizable.Graph layoutThe next step was to actually find a good way to arrange the data. We implemented a custom layout algorithm on top ofthe available algorithms from the yFiles diagramming library.To emphasize the affiliation of the characters to the houses, we made the algorithm place the house nodes in a circle and arrange the members of a house in the according sector.The families get placed on concentric circles around the center of the inner circle. Children are placed further outside than their parents while we tried to set spouses and lovers on the same circle. Characters that are associated to just one house, like servants, dragons or direwolves, are also assigned to the respective house.There are two characters (High Sparrow, Night King) located outside the sectors because they do not belong to any house and are only connected through a kill.There are three types of edges: family ties, allegiances, and killings. We used different edge routing algorithms for those edges so they became easily distinguishable. Edges that describe family relations or allegiances are straight lines when they appear inside a house sector. Inside the inner circle they get routed in arcs whose thicknesses represent the number of edges between two houses.Finally, the kill edges were quite a challenge. They connect characters from all parts of the diagram and there are many of them. After some experiments, we decided that they must run outside the inner circle. They were implemented as two straight segments and an arc. This way they can easily be distinguished from the other types of connections. Of course, we tried to minimize the number of crossings to make it easier to follow the edges.Styling the diagramCreating a clear arrangement of the characters and routing the connections between them was the first important step. But a diagram where nodes are only simple, colored shapes and edges are black lines with an arc is not really an eye-catcher. Using yFiles with its many styling options we were sure it would be easy and fun to change that situation. The nodes on the inner circle carry the emblem of the house or group they represent. They mark the sector as well as the end points for the inner edges.Characters are visualized using a picture of a person or animal along with their name. The border shows the color ofthe house to which the character belongs by birth.Since there are three different kinds of edges, we decided to give them different colors. Edges that represent family relationships are drawn in the color of the family’s house. In case those edges connect to two house nodes, theychange color in the middle. Allegiance edges are drawn using dark blue, dotted lines, independent of where the edge is located.As previously mentioned, kill edges are routed in different style. To separate them even more from the other edges, they have a dark red color. Their direction, i.e. who killed who, is indicated by port icons. The source is marked with a bloody knife while at the target a skull is shown.The final resultBelow you find the final diagram created by our algorithm which combines all described features.So now that we have a concise layout of the data and also a nice visualization of the diagram elements, what can we learn from the visualization? We think that the diagram provides a lot of information at a glance. Important characters can easily be recognized by their size and by the amount of relations they have. The large, most important houses immediately stand out.Even though someone who is very familiar with ‘Game of Thrones’ will probably not learn something totally new, our diagram makes it very easy and efficient to get an overview over the involved groups and characters.How do you like the results? Let us know! Just leave us a message in the social media or using our contact information.