Category Archives: Windows Azure

Running Neo4j on Windows Azure

Leave a reply

There are a number of approaches to getting Neo4j onto Azure, some of which are listed on the Neo4j’s website. Dynamic Deploy looked like the best approach but for some reason it would not offer deployment to Europe. There are also some pre-built Linux images but I wanted a Windows-based install so that I could easily troubleshoot any problems. Eventually I followed this post which installs Neo4j into an Azure worker role. This really shows what can be done with the PaaS approach, as opposed to using a VM: it installs and configures Java and Neo4j, creates and/or attaches a virtual hard drive and starts Neo4j bound to a custom port. As the author notes this was written against an older version of the SDK. To bring it to the latest version just search out all the references to Azure libraries and change then to the latest version. I also had problems with the VM as it seems the connection can’t be https; I’ve not investigated why this is so for now I’ve changed it, as shown below, to http:

Neo4j Azure Connection String B

The Storage connection string setting should now look like this:

Neo4j Azure Connection String A

And one more thing, you may find firewalls block port 5000 so you could try some commonly allowed ones like 8080.

 

Email, RSS Follow

Circular Layouts, Binning and Edge Bundling

Leave a reply

Circular layouts can be very useful and NodeXL does a great job of rapidly exploring social network graphs. However I needed to generate graphs programmatically so turned to the NodeXL libraries. I picked the circular layout but was disappointed with what it produced:

circular plain

Let me explain what this diagram is attempting to show: the rectangular labels are the organisation’s tweeters, the colours represent a sub-division of the organisation they work for. Unlabelled nodes (circles) are the followers. Followers are coloured green or orange based on which information the organisations would like them to receive and this colouring is the same for two specialist sub-divisions. Where an orange line is seen going to a green follower, or vice-versa, then it can be implied that the follower is not receiving the desired information.

This layout is not great: the order of nodes is just as they were added to the graph: all the organisations’ tweeters were added first so cluster together and there is no logic to the order the remaining nodes (followers) are added.

The first improvement that can be made is to use binning. See  http://www.smrfoundation.org/2010/01/14/component-binning-a-network-layout-improvement-in-nodexl-v-108/

To apply Binning with the NodeXL library simply set the UseBinning property:

oCircleLayout.LayoutStyle = LayoutStyle.UseBinning;

The layout now looks like this:

circular binning

We can see now that there is a cluster of followers who follow multiple tweeters from the organisation (clustered towards the top). However it is still quite confusing where a lot of lines cross-over. Maybe curved lines would be better….

oNodeXLVisual.GraphDrawer.EdgeDrawer.CurveStyle = EdgeCurveStyle.Bezier;

circular bezier

Not really an improvement. The answer is Edge Bundling, see these links for better explanations than I can provide:

http://www.cg.tuwien.ac.at/courses/InfoVis/HallOfFame/2007/Alsallakh/

http://www.win.tue.nl/~dholten/papers/bundles_infovis.pdf

http://www.infosthetics.com/archives/2009/06/force_directed_edge_bundling_for_graph_visualization.html

this is how to add it from the NodeXL libraries:

EdgeBundler ebl = new EdgeBundler();

ebl.UseThreading = false; // when running in Azure or it hangs

ebl.BundleAllEdges(oGraph, new Rectangle(0, 0, GraphWidth, GraphHeight));

oNodeXLVisual.GraphDrawer.EdgeDrawer.CurveStyle = EdgeCurveStyle.CurveThroughIntermediatePoints; 

and the result:

circular bundling

which I hope you’ll agree helps reveal some real structure about the tweeters and
their followers.

Update 15/07/2013

I’ve created a test page if you want to try out these features of NodeXL here or get NodeXL

Email, RSS Follow

Mining Twitter from Windows Azure (Part 1)

2 Replies

Organisations need to know what is being said about them, and Twitter is one of the obvious places to find this out. I’m not looking at trawling all of Twitter but targeting tweets about the organisation or its competitors and what the organisation and its competitors are tweeting.

Because Twitter data is mostly already public, and no private data is being retrieved, this investigation was also a good opportunity to explore Microsoft’s Azure cloud computing platform.

I have built the architecture shown in the following diagram in order to collect and analyse Twitter data:

Twitter Monitor

Nothing here is not well documented elsewhere but I’ll run through the key components:

Having built the above I can say it all works very well, my key learning point are:

  • The Twitter API provides a search which is a great place to start exploring who is saying what about an organisations
  • Thinking in terms of relational database tables is probably a mistake (that I have fallen into). For example I have a table of followers for a given Twitter account and then a table of information about Twitter accounts; to get the information for all the followers of a given Twitter account it is necessary to join these table. Doh! Can’t do that with Azure Table Storage. Now because I’m not dealing with large volumes this can be accomplished by ‘joining’ the tables in memory (I used a hash table but I’m sure LINQ can do it). The correct solution is either to use SQL Azure or move the information about the Twitter accounts into the table of followers and accept some duplication…or maybe a graph database 🙂
  • Probably this is a bug: TweetSharp (or the underlying JSON and REST libraries) will cause a stack overflow (!) if trying to retrieve a list of follower IDs when the Twitter user has protected tweets;  you can ensure this does not happen by checking the user’s protected status first.

 

Email, RSS Follow