Microsoft & .NET.NETImplementing Visual and Data Mashups Online

Implementing Visual and Data Mashups Online content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

What Are Online Web 2.0 Mashups?

In my previous article, “Web and Enterprise Mashups for Web Services and Data,” I covered different tools for the enterprise mashup creation, including web services mashups and data aggregation mashups. I also have shown how tools such as Aqua Logic can facilitate web services mashups. These tools are geared to provide a new breed of enterprise services and to ease business application development. But, these tools do not work with some popular communication protocols, such as RSS. In this article, I will discuss online mashups, which are not yet designed for the enterprise but could lead to new and exciting online applications. This, in turn, can lead to new businesses based on the ideas or implementations that arise from these application and services.

When culinary master chefs compete on the Iron Chef TV show, they are given a lot of ingredients and one secret ingredient to be used in all the dishes. They also are given an entire professional kitchen to prepare their sumptuous creations. The final dishes then are judged by a small panel of experts who get to comment on taste, texture, presentation, and overall quality. The viewers get to see the whole process but don’t get to participate in it, nor do they get to try the food.

Imagine for a moment that all of the millions of TV viewers would get to participate, and would get the same access to the same ingredients and same professional kitchen and its equipment. On top of that, whatever one viewer creates can be taken and improved by another viewer and then the entire community of viewers would get to taste and judge the creations.

Even though the master chefs have years of experience in the culinary arts, if they compete against the “community” they would not be able to come up with what the majority of people exactly want. The same concept is behind online web mashups, but instead of food, the ingredients are much less caloric, and much more data and solution centric. The idea of the millions of developers creating something together online practically guarantees that whatever will survive the judgment of the masses will be the most useful and popular application.

Many large software companies jumped on this idea and developed environments where developers can collaborate on combining content, functionality, and APIs to create new applications and services. These environments also let people share and publish whatever they create. Anything from simple apps for personal use to complex applications and services can be created in these environments. These apps then undergo process of elimination, improvement, and evolution by the whole community. The concept of online mashups is in line with the web 2.0 ideology as its very social in nature.

Mashup Development Environments

The online mashups would not be possible without advances in the modern web technology standards and technologies. Communication protocols and data exchange standards, in particular, act as a corner stone for the modern Mashup development. All of the mashups environments are ether downloadable browser extensions, downloadable software, or web based solutions. The technology behind these tools and sites is different, but they all rely on HTTP, RSS, SOAP and other standards to exchange data, call APIs and communicate with other mashups. They all allow some form of editor to create the combination of data, services or APIs and then some form of test interface and publishing interface.

Tools and Browser Extensions

Intel MashMaker is an example of a browser plug-in for Mashup development. This browser extension installs in Firefox or IE and runs the machine. It’s a visual tool, for data and presentation mashups, where all of the mashups are shareable and appear as composite widgets inside of the browser window. Essentially, the pages are modified to include an xtra layer of information on top of the actual page rendered in the browser. After installing the plug-in, Mashmaker will analyze sites being viewed for any possible technologies it can understand (web content, videos, maps, RSS feeds, photos) and allows users to manage the underlying data and combine it with other data sources. Afterwards, it can store the compilations, share them, and let others vote on them. It also allows users to discover new mashups. For example, the tool can find geolocation information on a popular social networking site (like Facebook), combine it with the Google Map widget, and show the location of your friends visually on the map, with their pictures. To do so, it uses a collaboratively edited shared database of extractors to obtain meaning from pages on the web.

Among all other downloadable tools, MashMaker is the most refined. It has a good collection of pre-build widgets, good documentation, and examples. Conceptually, it’s also very different then the other tools; it can discover data sources and change the way a page is presented to the user based on them.

Another example of a downloadable package is Serena Business Mashups (which together with Serena Mashup Composer), provides developers with a point-and-click environment for business-related mashups creation. The package essentially is a web application running on an app server. The mashups developed with Serena will run in an application container deployed to an app server and combine content and processes from many different systems, using industry-standard SOA and Web services interfaces.

Online only mashup creators

The two most popular online mashup sites are Yahoo Pipes and Microsoft Popfly. Google Mashup Editor is also gaining popularity, but it’s currently in a beta state, and has only a handful of features. Yahoo Pipes is developed in HTML and JavaScript and supports IE 7 and Firefox only. MS Popfly is based on Silverlight technology, which is somewhat similar to Flash.

Yahoo Pipes is designed primarily for visual creation of data mashups. It can take any data source, such as a CVS file, RSS feed, search engine output, url output, and so forth, and apply certain manipulation on the data to get the final output “through the pipe” in the way that the user wants. An example would be an RSS feed in French, translated into English dynamically, or a combination of two news RSS feeds into one and containing only terms that the user wants. Yahoo Pipes is the most polished and stable from the current online mashup offerings. However, it does have some limitations. For instance, the majority of mashups done with Yahoo are done around some sort of data feed because it is the easiest way to get data into the pipe.

Yahoo Pipes also has a collection of predefined operations that can be applied to the data source, such as extract of geo information, sorting, splitting, regex, and so on. But, it does not have a robust collection of predefined widgets like MS Popfly or MashMaker does. I could not find a way to extend the default widget set or to do a simple operation, such as an intersect of two feeds—a scenario where the pipe would output only items that are common in both feeds. This means that users can use only what Yahoo is providing and this somewhat limits the benefit of the application.

Here are screen shots of Yahoo Pipes:

Figure 1: Yahoo pipes mashup editor

If Yahoo Pipes focuses on manipulation of raw data in “pipes,” Microsoft Popfly focuses on visual manipulation of widgets to create new and unique services and applications. Mashups developed with Popfly can be saved and shared, and it also has very good online help.

MS Popfly is another very popular online application, and it offers a robust environment for manipulating “building blocks” that make up mashups. These blocks are essentially widgets that have some logic on how to get data from RSS feed, call a Web service API, or some other data source. Many of the blocks require a special API access key; this means that Microsoft developed partnerships with many popular sites, or made use of what APIs are already available. For example, keys are required for Facebook and Technorati, and can be generated through their APIs.

Figure 2: MS Popfly mashup editor

Google Editor is much more developer centric. It does not have a visual interface like Pipes or Popfly do, nor does it have a library of predefined widgets. The only interface it has is an editor for manual mashup coding.

Figure 3: Google Mashup Editor

Developing Mashups with Yahoo Pipes

So, what does it take to create a mashup online? The answer depends on the tool or site you will use. When using Intel Mashmaker, Yahoo Pipes, or MS Popfly, fairly sophisticated environments will let you visually compose by dragging and dropping various data sources, RSS feeds, and widgets. When using Google Editor or Serena Mashup Composer, a little more work and higher learning curve will be required.

Look at how to make a new mashup with Yahoo pipes. In this example, I’ll be combining output from two news feeds (CNN Top Stories and Reuters Top Stories) and filtering them on the search keyword.

When you log in into the editor, all of the pipe building widgets and operations are on the left panel. To add a new data source, simply drag a corresponding widget onto the grid.

Figure 4: Step 1—New Yahoo Pipe

For this article, I have used two popular RSS news feeds, Reuters and CNN: and Enter the feed URL and click on the Refresh link in the debug pane on the bottom. It will show the output from the pipe.

Repeat the same for the CNN feed.

Add a “Union” operation widget to combine the two news sources, and join the output from the widget by dragging a connecting wire between them.

Figure 5: Step 2—Two sources and union widget

Add another operation to output only unique items, in case there are two identical titles (this is probably unlikely in this case, but could be useful with other sources).

Figure 6: Step 3—Two sources and union widget and Unique widget

Add the Filter widget; it will filter output for a particular keyword. This widget is configured to accept text from the text-field on the main page where this whole pipe will be executed. It can have a hard-coded value for testing, but dynamic filtering is much more useful. Finally, connect the Filter to the pipe output widget.

Figure 7: Step 4—Complete Pipe

Save the pipe and click the Run pipe option. Your Yahoo Pipes application will create a special page automatically (with its own unique URL), where the pipe can be executed, and its output viewed. Yahoo Pipes can even email periodic alerts (with the result) or create a new RSS feed out of the output (which can be reused in yet another pipe mashup).

Figure 8: Pipe run page and output


In this article, I have discussed various online and offline tools for data mashups. These tools are a powerful way to create new applications online and open vast possibilities for service developers. In my previous article, Web and Enterprise Mashups for Web Services and Data.” I talked about enterprise mashups, and even though this wave of online data mashups and enterprise mashup tools is in its early stages, it’s already showing a lot of promise.

Most of the current tools have limitations. They all show some interface bugs and design issues, but once these are worked out, these tools will bring a lot of value to the end users and online communities in general. It will be very interesting to see what unique applications will appear from the mashup world.


Online Resources

About the Author

Vlad Kofman works on enterprise-scale projects for the major Wall Street firms. He has also worked on the defense contracts for the U.S. government. His main interests are object-oriented programming methodologies, UI, and design patterns.

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Latest Posts

Related Stories